Interface IMapper
Turns an input string into a sequence of token ids. This is the Hugging Face equivalent of Models.
Namespace: Unity.InferenceEngine.Tokenization.Mappers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public interface IMapperMethods
IdToToken(int)
Gets the token value from the specified id.
Declaration
[CanBeNull]
string IdToToken(int id)Parameters
| Type | Name | Description |
|---|---|---|
| int | id | The ID of the requested token. |
Returns
| Type | Description |
|---|---|
| string | The token value. |
TokenToId(string, out int)
Gets the ID of the specified token
Declaration
bool TokenToId(string token, out int id)Parameters
| Type | Name | Description |
|---|---|---|
| string | token | The token we want to get the ID of. |
| int | id | The ID of the specified |
Returns
| Type | Description |
|---|---|
| bool | Whether the token exists. |
Tokenize(IReadOnlyList<SubString>, Output<Token>)
Tokenizes a list of string values.
Declaration
void Tokenize(IReadOnlyList<SubString> input, Output<Token> output)Parameters
| Type | Name | Description |
|---|---|---|
| IReadOnlyList<SubString> | input | The list of string values to tokenize. |
| Output<Token> | output | The recipient of the converted tokens. |