Interface ITruncator

Splits sequences of tokens into smaller collection of sequences.

Namespace: Unity.InferenceEngine.Tokenization.Truncators

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public interface ITruncator

Methods

Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Splits sequences of tokens into smaller collection of sequences.

Declaration

void Truncate(IReadOnlyList<Token> inputA, IReadOnlyList<Token> inputB, int numAddedTokens, Output<IEnumerable<Token>> outputA, Output<IEnumerable<Token>> outputB)

Parameters

Type	Name	Description
IReadOnlyList<Token>	inputA	The primary sequence of tokens (mandatory).
IReadOnlyList<Token>	inputB	The optional secondary sequence of tokens.
int	numAddedTokens	The number of tokens that the IPostProcessor steps will add.
Output<IEnumerable<Token>>	outputA	The target container of the truncated subsequences of inputA.
Output<IEnumerable<Token>>	outputB	The target container of the truncated subsequences of inputB.