Enum SplitDelimiterBehavior
Options for how to deal with the delimiter when splitting the input string. See RegexSplitPreTokenizer
Namespace: Unity.InferenceEngine.Tokenization
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public enum SplitDelimiterBehaviorFields
| Name | Description |
|---|---|
| Contiguous | Variation of Isolated, but merges the contiguous delimiters. |
| Isolated | The delimiter is kept as a separate token. |
| MergedWithNext | The delimiter is prepended to the next token. |
| MergedWithPrevious | The delimiter is appended to the previous token. |
| Removed | The delimiter is not included in the output tokens at all. |