Namespace Unity.InferenceEngine.Tokenization.Decoders

Classes

ByteFallbackDecoder

Converts tokens looking like "<0x61>" to character, and attempts to concatenate them into a string. If the tokens cannot be decoded, '�' is used instead for each inconvertible byte token.

ByteLevelDecoder

Converts byte-level characters to unicode characters, then concat them into a single string.

DefaultDecoder

Default decoder. Does not change the input chunks.

FuseDecoder

Fuse Decoder combine the tokens in list into a single large token.

ReplaceDecoder

Replaces a string pattern from the tokens in the list.

StripDecoder

Strip Decoder removes certain char from the substring of the token in the list.

WordPieceDecoder

An implementation of the WordPiece decoding algorithm.

Interfaces

IDecoder

Applies modifications to the input detokenized strings.