Class SequencePreTokenizer

Applies a sequence of pre tokenizers.

Inheritance
object
SequencePreTokenizer
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class SequencePreTokenizer : IPreTokenizer

Constructors

SequencePreTokenizer(params IPreTokenizer[])

Initializes a new instance of the SequencePreTokenizer type.

Declaration
public SequencePreTokenizer(params IPreTokenizer[] preTokenizers)
Parameters
Type Name Description
IPreTokenizer[] preTokenizers

Sequence of pre-tokenizers to apply.
Exceptions
Type Condition
ArgumentNullException

preTokenizers cannot be null.
ArgumentException

preTokenizers cannot be empty.

Methods

PreTokenize(SubString, Output<SubString>)

Pre-cuts the input into smaller parts.

Declaration
public void PreTokenize(SubString input, Output<SubString> output)
Parameters
Type Name Description
SubString input

The source to pre-cut.
Output<SubString> output

Target collection of generated pre-tokenized strings.

Implements