Class LongestFirstTruncator

This truncation strategy truncates the longest sequence of tokens first. In case a secondary sequence of tokens is not provided, it doesn't remove any token.

Inheritance

object

StrategicTruncator

LongestFirstTruncator

Implements

ITruncator

Inherited Members

StrategicTruncator.Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

StrategicTruncator.GetRanges(int, int, Output<Range>)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Truncators

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class LongestFirstTruncator : StrategicTruncator, ITruncator

Constructors

LongestFirstTruncator(IRangeGenerator, int, int)

Initializes a new instance of the StrategicTruncator type.

Declaration

public LongestFirstTruncator(IRangeGenerator rangeGenerator, int maxLength, int stride)

Parameters

Type	Name	Description
IRangeGenerator	rangeGenerator	Generates the range of each truncated sequence.
int	maxLength	The maximum length of each truncated sequence.
int	stride	How to go along the sequence of tokens.

Methods

Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Truncates the input sequences of tokens (tokensA and tokensB) and passes the resulting truncated subsequences to outputA and outputB.

Declaration

protected override void Truncate(IReadOnlyList<Token> tokensA, IReadOnlyList<Token> tokensB, int maxLength, int toRemove, Output<IEnumerable<Token>> outputA, Output<IEnumerable<Token>> outputB)

Parameters

Type	Name	Description
IReadOnlyList<Token>	tokensA	The tokens of the primary sequence.
IReadOnlyList<Token>	tokensB	The tokens of the secondary sequence.
int	maxLength	The maximum length of each truncated sequence.
int	toRemove	The total number of tokens to remove from tokensA and tokensB in order to get the first truncated sequence.
Output<IEnumerable<Token>>	outputA	The target container for truncated subsequences from tokensA.
Output<IEnumerable<Token>>	outputB	The target container for truncated subsequences from tokensB.

Overrides

StrategicTruncator.Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Implements

ITruncator