Class UnicodeNormalizer
Applies standard Unicode normalization.
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.Normalizers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class UnicodeNormalizer : INormalizerConstructors
UnicodeNormalizer(NormalizationForm)
Initializes a new instance of the UnicodeNormalizer type.
Declaration
public UnicodeNormalizer(NormalizationForm form = NormalizationForm.FormC)Parameters
| Type | Name | Description |
|---|---|---|
| NormalizationForm | form | The standard unicode normalization form. |
Methods
Normalize(SubString)
Applies transformations to the input string before pre-tokenization.
Declaration
public SubString Normalize(SubString input)Parameters
| Type | Name | Description |
|---|---|---|
| SubString | input | The string to transform. |
Returns
| Type | Description |
|---|---|
| SubString | The resulting string. |