Class HfPreTokenizerAttribute
Identifies the parser for an implementation of IPreTokenizer
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.Parsers.HuggingFace
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = false)]
public class HfPreTokenizerAttribute : HfAttributeConstructors
HfPreTokenizerAttribute(string)
Initializes a new instance of the HfPreTokenizerAttribute type.
Declaration
public HfPreTokenizerAttribute(string type)Parameters
| Type | Name | Description |
|---|---|---|
| string | type | The type name of the pre tokenizer. It corresponds to the name in the Hugging Face JSON configuration. |
Fields
Type
The type of decoder.
Declaration
public readonly string TypeField Value
| Type | Description |
|---|---|
| string |