Class RuneSplitPreTokenizer

Splits the input by the runes.

Inheritance
object
RuneSplitPreTokenizer
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class RuneSplitPreTokenizer : IPreTokenizer

Constructors

RuneSplitPreTokenizer(SplitDelimiterBehavior, bool)

Initializes a new instance of the RuneSplitPreTokenizer type.

Declaration
public RuneSplitPreTokenizer(SplitDelimiterBehavior behavior, bool invert = false)
Parameters
Type Name Description
SplitDelimiterBehavior behavior

What to do with substrings matching a rune.
bool invert

Inverts the pattern matching.

Methods

PreTokenize(SubString, Output<SubString>)

Pre-cuts the input into smaller parts.

Declaration
public void PreTokenize(SubString input, Output<SubString> output)
Parameters
Type Name Description
SubString input

The source to pre-cut.
Output<SubString> output

Target collection of generated pre-tokenized strings.

Implements