Namespace Unity.InferenceEngine.Tokenization.Normalizers

Classes

AppendNormalizer

Adds a suffix to the input string.

BertNormalizer

Normalizes raw text input for Bert model.

DefaultNormalizer

Does not apply any transformation.

PrependNormalizer

Adds a prefix to the input string.

RegexReplaceNormalizer

Replaces a specified pattern by another string.

ReplaceNormalizer

Replaces a specified pattern by another string.

SequenceNormalizer

Applies multiple INormalizer.

UnicodeNormalizer

Applies standard Unicode normalization.

Interfaces

INormalizer

Applies transformations to the input string before pre-tokenization.