Class StringSplitPreTokenizer

Splits the input based on a string pattern.

Inheritance
object
StringSplitPreTokenizer
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class StringSplitPreTokenizer : IPreTokenizer

Constructors

StringSplitPreTokenizer(string, SplitDelimiterBehavior, bool)

Initializes a new instance of the StringSplitPreTokenizer type.

Declaration
public StringSplitPreTokenizer(string pattern, SplitDelimiterBehavior behavior, bool invert = false)
Parameters
Type Name Description
string pattern

The pattern on which the input string is split.
SplitDelimiterBehavior behavior

Indicates how to handle splits and patterns. SplitDelimiterBehavior
bool invert

Whether of not to invert the pattern. Not yet implemented.

Methods

PreTokenize(SubString, Output<SubString>)

Pre-cuts the input into smaller parts.

Declaration
public void PreTokenize(SubString input, Output<SubString> output)
Parameters
Type Name Description
SubString input

The source to pre-cut.
Output<SubString> output

Target collection of generated pre-tokenized strings.

Implements