Class RegexSplitPreTokenizer

Splits the input based on a regular expression.

Inheritance

object

RegexSplitPreTokenizer

Implements

IPreTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class RegexSplitPreTokenizer : IPreTokenizer

Constructors

RegexSplitPreTokenizer(string, SplitDelimiterBehavior, bool)

Initializes a new instance of the RegexSplitPreTokenizer type.

Declaration

public RegexSplitPreTokenizer(string pattern, SplitDelimiterBehavior behavior, bool invert = false)

Parameters

Type	Name	Description
string	pattern	The pattern on which the input string is split.
SplitDelimiterBehavior	behavior	Indicates how to handle splits and patterns. SplitDelimiterBehavior
bool	invert	Whether of not to invert the pattern. Not yet implemented.

RegexSplitPreTokenizer(Regex, SplitDelimiterBehavior, bool)

Initializes a new instance of the RegexSplitPreTokenizer type.

Declaration

public RegexSplitPreTokenizer(Regex pattern, SplitDelimiterBehavior behavior, bool invert = false)

Parameters

Type	Name	Description
Regex	pattern	The pattern on which the input string is split.
SplitDelimiterBehavior	behavior	Indicates how to handle splits and patterns. SplitDelimiterBehavior
bool	invert	Whether of not to invert the pattern. Not yet implemented.

Methods

PreTokenize(SubString, Output<SubString>)

Pre-cuts the input into smaller parts.

Declaration

public void PreTokenize(SubString input, Output<SubString> output)

Parameters

Type	Name	Description
SubString	input	The source to pre-cut.
Output<SubString>	output	Target collection of generated pre-tokenized strings.

Implements

IPreTokenizer