Namespace Unity.InferenceEngine.Tokenization.SplitDelimiterBehaviors

Aggregates all the successive delimiters into a single split and keeps all the content splits as is.

Keeps both the content splits and the delimiter splits, separated.

Merges each single delimiter split with its preceding content split.

Merges each single delimiter split with its next content split.

Removes the delimiter splits, only the content splits only.