Namespace Unity.InferenceEngine.Tokenization.Mappers

Classes

BpeMapper

Turns a string input into a sequence of Token instances using the Byte-Pair Encoding strategy.

WordPieceMapper

Turns an input string into a sequence of token ids using the Word Piece strategy.

Structs

BpeMapperOptions

Configuration settings for the Byte Pair Encoding (BPE) mapper used in tokenization.

Represents a mergeable pair of token values used in Byte Pair Encoding (BPE) tokenization. Each pair consists of two consecutive token strings that can be merged into a single token during the BPE encoding process. See BpeMapper.

Interfaces

IMapper

Turns an input string into a sequence of token ids. This is the Hugging Face equivalent of Models.