Class DefaultPreTokenizer

Default placeholder implementation of a pre-tokenizer. Does not pre-cut the input.

Inheritance

object

DefaultPreTokenizer

Implements

IPreTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.PreTokenizers

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class DefaultPreTokenizer : IPreTokenizer

Methods

PreTokenize(SubString, Output<SubString>)

Pre-cuts the input into smaller parts.

Declaration

public void PreTokenize(SubString input, Output<SubString> output)

Parameters

Type	Name	Description
SubString	input	The source to pre-cut.
Output<SubString>	output	Target collection of generated pre-tokenized strings.

Implements

IPreTokenizer