Class WordLevelMapper
A word-level tokenization mapper that converts between tokens and their corresponding IDs.
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.Mappers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class WordLevelMapper : IMapperConstructors
WordLevelMapper(Dictionary<string, int>, string)
Initializes a new instance of the WordLevelMapper class.
Declaration
public WordLevelMapper(Dictionary<string, int> vocab, string unkToken)Parameters
| Type | Name | Description |
|---|---|---|
| Dictionary<string, int> | vocab | The vocabulary dictionary mapping token strings to their corresponding IDs. |
| string | unkToken | The unknown token string used when a token is not found in the vocabulary. |
Methods
IdToToken(int)
Gets the token value from the specified id.
Declaration
public string IdToToken(int id)Parameters
| Type | Name | Description |
|---|---|---|
| int | id | The ID of the requested token. |
Returns
| Type | Description |
|---|---|
| string | The token value. |
TokenToId(string, out int)
Gets the ID of the specified token
Declaration
public bool TokenToId(string token, out int id)Parameters
| Type | Name | Description |
|---|---|---|
| string | token | The token we want to get the ID of. |
| int | id | The ID of the specified |
Returns
| Type | Description |
|---|---|
| bool | Whether the token exists. |
Tokenize(IReadOnlyList<SubString>, Output<Token>)
Tokenizes a list of string values.
Declaration
public void Tokenize(IReadOnlyList<SubString> input, Output<Token> output)Parameters
| Type | Name | Description |
|---|---|---|
| IReadOnlyList<SubString> | input | The list of string values to tokenize. |
| Output<Token> | output | The recipient of the converted tokens. |