Class Encoding
Contains the result of a tokenization pipeline ran by a Tokenizer instance.
Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public class Encoding : IEncodingProperties
Length
The number of tokens.
Declaration
public int Length { get; }Property Value
| Type | Description |
|---|---|
| int |
Overflow
In case the tokenization pipeline produces more tokens than the expected size, the following tokens are stored into another IEncoding instance. This overflow can also define its own overflow, similarly to a linked list.
Declaration
public IEncoding Overflow { get; }Property Value
| Type | Description |
|---|---|
| IEncoding |
Methods
GetAttentionMask(ICollection<int>)
The attention mask. When a tokenization requires truncation and padding, this mask indicates which tokens are the most relevant.
Declaration
public int GetAttentionMask(ICollection<int> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<int> | output | The target container of attention state. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetIds(ICollection<int>)
The list of token ids.
Declaration
public int GetIds(ICollection<int> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<int> | output | The target container of ids. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetOffsets(ICollection<Range>)
The token offsets.
Declaration
public int GetOffsets(ICollection<Range> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<Range> | output | The target container of offsets. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetSpecialMask(ICollection<int>)
The special tokens mask
Declaration
public int GetSpecialMask(ICollection<int> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<int> | output | The target container of special states. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetTokens(ICollection<Token>)
The list of tokens.
Declaration
public int GetTokens(ICollection<Token> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<Token> | output | The target container of tokens. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetTypeIds(ICollection<int>)
The type ids.
Declaration
public int GetTypeIds(ICollection<int> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<int> | output | The target container of type ids. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |
GetValues(ICollection<string>)
The list of token ids.
Declaration
public int GetValues(ICollection<string> output)Parameters
| Type | Name | Description |
|---|---|---|
| ICollection<string> | output | The target container of values. |
Returns
| Type | Description |
|---|---|
| int | The number of available tokens. |