Class NmtNormalizer

Provides NMT (Neural Machine Translation) normalization for text preprocessing. Filters out control characters and normalizes various whitespace and special Unicode characters to standard spaces.

Inheritance

object

NmtNormalizer

Implements

INormalizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Normalizers

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class NmtNormalizer : INormalizer

Methods

Normalize(SubString)

Applies transformations to the input string before pre-tokenization.

Declaration

public SubString Normalize(SubString input)

Parameters

Type	Name	Description
SubString	input	The string to transform.

Returns

Type	Description
SubString	The resulting string.

Implements

INormalizer