Class LongestFirstStrategy

Truncation strategy that iteratively truncates the longest sequence first. This strategy attempts to balance the lengths of both sequences by removing tokens from the longer sequence until the combined length fits within the maximum allowed length.

Inheritance

object

LongestFirstStrategy

Implements

ITruncationStrategy

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Truncators.Strategies

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class LongestFirstStrategy : ITruncationStrategy

Properties

Instance

Gets the singleton instance of the LongestFirstStrategy.

Declaration

public static ITruncationStrategy Instance { get; }

Property Value

Type	Description
ITruncationStrategy

Methods

GetTruncationLength(int, IReadOnlyList<Token>, IReadOnlyList<Token>)

Calculates the truncation lengths for two token sequences by truncating the longest sequence first. If both sequences need truncation, they will be balanced to approximately equal lengths.

Declaration

public (int lengthA, int lengthB) GetTruncationLength(int maxLength, IReadOnlyList<Token> tokensA, IReadOnlyList<Token> tokensB)

Parameters

Type	Name	Description
int	maxLength	The maximum combined length allowed for both sequences.
IReadOnlyList<Token>	tokensA	The first sequence of tokens.
IReadOnlyList<Token>	tokensB	The second sequence of tokens (optional).

Returns

Type	Description
(int, int)	A tuple containing the truncated lengths for both sequences. When significant truncation is needed, both sequences will be truncated to approximately half of the maximum length.

Implements

ITruncationStrategy