Namespace Unity.InferenceEngine.Tokenization.Truncators

Classes

DefaultTruncator

Placeholder implementation of the truncation. Does not truncate, only concatenates the primary and secondary sequence of tokens.

LeftDirectionRangeGenerator

Generates a sequence of Range starting from the left (0, the lower bound of the source).

LongestFirstTruncator

This truncation strategy truncates the longest sequence of tokens first. In case a secondary sequence of tokens is not provided, it doesn't remove any token.

RangeGeneratorBase

Base implementation of the builtin IRangeGenerator. Runs the necessary parameter validation before calling GetRangesInternal(int, int, int, Output<Range>).

RightDirectionRangeGenerator

Generates a sequence of Range starting from the right (the upper bound of the source).

StrategicTruncator

Base implementation for builtin ITruncator types using IRangeGenerator to truncate the sequences of tokens.

Interfaces

IRangeGenerator

Generates a sequence of Ranges.

ITruncator

Splits sequences of tokens into smaller collection of sequences.