Namespace Unity.InferenceEngine.Tokenization.Truncators.Strategies

Classes

LongestFirstStrategy

Truncation strategy that iteratively truncates the longest sequence first. This strategy attempts to balance the lengths of both sequences by removing tokens from the longer sequence until the combined length fits within the maximum allowed length.

OnlyFirstStrategy

Truncation strategy that only truncates the first sequence. The second sequence length is preserved, and the first sequence is adjusted to fit within the maximum length.

OnlySecondStrategy

Truncation strategy that only truncates the second sequence. The first sequence length is preserved, and the second sequence is adjusted to fit within the maximum length.

Interfaces

ITruncationStrategy

Defines a strategy for truncating token sequences to fit within a maximum length constraint. Implementations determine how to distribute the available length between two token sequences.