Namespace Unity.InferenceEngine.Tokenization.PostProcessors

Classes

ByteLevelPostProcessor

ByteLevel post processor only concatenates the pair sequences. The former implementation from Hugging Face trims offsets of tokenized strings, but this implementation does support offsets.

DefaultPostProcessor

Interlaces the primary and secondary sequences of tokens.

RobertaPostProcessor

Adds the special tokens needed by a Roberta model. Surrounds the single sequence with CLS and SEP tokens. Surrounds the second sequence of a pair and SEP tokens.

TemplatePostProcessor

Post processor using the templating approach.

Interfaces

IPostProcessor

Transforms the sequences of tokens from the truncated output of IMapper and merges it into a single sequence.