Interface IPreTokenizer
Pre-cuts the input string into smaller parts. Those parts will be passed to the IMapper for tokenization.
Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public interface IPreTokenizerMethods
PreTokenize(SubString, Output<SubString>)
Pre-cuts the input into smaller parts.
Declaration
void PreTokenize(SubString input, Output<SubString> output)Parameters
| Type | Name | Description |
|---|---|---|
| SubString | input | The source to pre-cut. |
| Output<SubString> | output | Target collection of generated pre-tokenized strings. |