Class GenericTruncator

General implementation of truncation, replacing the obsolete StrategicTruncator and LongestFirstStrategy while supporting more options.

Inheritance

object

GenericTruncator

Implements

ITruncator

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Truncators

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class GenericTruncator : ITruncator

Constructors

GenericTruncator(ITruncationStrategy, IRangeGenerator, int, int)

Initializes a new instance of the GenericTruncator type.

Declaration

public GenericTruncator(ITruncationStrategy strategy, IRangeGenerator direction, int maxLength, int stride)

Parameters

Type	Name	Description
ITruncationStrategy	strategy	The strategy of truncation. LongestFirstStrategy, OnlyFirstStrategy and OnlySecondStrategy are the built-in implementations.
IRangeGenerator	direction	The truncation direction, either Left with LeftDirectionRangeGenerator or Right with RightDirectionRangeGenerator.
int	maxLength	The maximum length of each truncated sequence.
int	stride	How to go along the sequence of tokens.

Methods

Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Splits sequences of tokens into smaller collection of sequences.

Declaration

public void Truncate(IReadOnlyList<Token> inputA, IReadOnlyList<Token> inputB, int numAddedTokens, Output<IEnumerable<Token>> outputA, Output<IEnumerable<Token>> outputB)

Parameters

Type	Name	Description
IReadOnlyList<Token>	inputA	The primary sequence of tokens (mandatory).
IReadOnlyList<Token>	inputB	The optional secondary sequence of tokens.
int	numAddedTokens	The number of tokens that the IPostProcessor steps will add.
Output<IEnumerable<Token>>	outputA	The target container of the truncated subsequences of `inputA`.
Output<IEnumerable<Token>>	outputB	The target container of the truncated subsequences of `inputB`.

Implements

ITruncator