Namespace Unity.InferenceEngine.Tokenization.Parsers.HuggingFace

Classes

HfAttribute

Base type for Hugging Face component parsing attributes.

HfDecoderAttribute

Identifies the parser for an implementation of IDecoder

HfModelAttribute

Identifies the parser for an implementation of IMapper

HfNormalizerAttribute

Identifies the parser for an implementation of INormalizer

HfPaddingAttribute

Identifies the parser for an implementation of IPadding

HfPostProcessorAttribute

Identifies the parser for an implementation of IPostProcessor

HfPreTokenizerAttribute

Identifies the parser for an implementation of IPreTokenizer

HfTruncationAttribute

Identifies the parser for an implementation of ITruncator

HuggingFaceParser

Parses a Hugging Face JSON configuration and builds a Tokenizer.

Interfaces

IComponentBuilder

High-level non-generic interface for Hugging Face tokenizer component builders.

IComponentBuilder<TComponent>

Generic Interface for Hugging Face tokenizer component builders.