Class StrategicTruncator

Base implementation for builtin ITruncator types using IRangeGenerator to truncate the sequences of tokens.

Inheritance

object

StrategicTruncator

LongestFirstTruncator

Implements

ITruncator

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Truncators

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public abstract class StrategicTruncator : ITruncator

Constructors

StrategicTruncator(IRangeGenerator, int, int)

Initializes a new instance of the StrategicTruncator type.

Declaration

protected StrategicTruncator(IRangeGenerator rangeGenerator, int maxLength, int stride)

Parameters

Type	Name	Description
IRangeGenerator	rangeGenerator	Generates the range of each truncated sequence.
int	maxLength	The maximum length of each truncated sequence.
int	stride	How to go along the sequence of tokens.

Methods

GetRanges(int, int, Output<Range>)

Gets the Ranges to truncate the sequences of tokens.

Declaration

protected int GetRanges(int length, int rangeMaxLength, Output<Range> output)

Parameters

Type	Name	Description
int	length	The length of the sequence of tokens.
int	rangeMaxLength	The maximum length of each chunk.
Output<Range>	output	The sequence of Range

Returns

Type	Description
int	The number of ranges generated.

Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Truncates the input sequences of tokens (tokensA and tokensB) and passes the resulting truncated subsequences to outputA and outputB.

Declaration

protected abstract void Truncate(IReadOnlyList<Token> tokensA, IReadOnlyList<Token> tokensB, int maxLength, int toRemove, Output<IEnumerable<Token>> outputA, Output<IEnumerable<Token>> outputB)

Parameters

Type	Name	Description
IReadOnlyList<Token>	tokensA	The tokens of the primary sequence.
IReadOnlyList<Token>	tokensB	The tokens of the secondary sequence.
int	maxLength	The maximum length of each truncated sequence.
int	toRemove	The total number of tokens to remove from tokensA and tokensB in order to get the first truncated sequence.
Output<IEnumerable<Token>>	outputA	The target container for truncated subsequences from tokensA.
Output<IEnumerable<Token>>	outputB	The target container for truncated subsequences from tokensB.

Truncate(IReadOnlyList<Token>, IReadOnlyList<Token>, int, Output<IEnumerable<Token>>, Output<IEnumerable<Token>>)

Splits sequences of tokens into smaller collection of sequences.

Declaration

public void Truncate(IReadOnlyList<Token> inputA, IReadOnlyList<Token> inputB, int numAddedTokens, Output<IEnumerable<Token>> outputA, Output<IEnumerable<Token>> outputB)

Parameters

Type	Name	Description
IReadOnlyList<Token>	inputA	The primary sequence of tokens (mandatory).
IReadOnlyList<Token>	inputB	The optional secondary sequence of tokens.
int	numAddedTokens	The number of tokens that the IPostProcessor steps will add.
Output<IEnumerable<Token>>	outputA	The target container of the truncated subsequences of inputA.
Output<IEnumerable<Token>>	outputB	The target container of the truncated subsequences of inputB.

Exceptions

Type	Condition
ArgumentNullException	inputA is null. outputA is null. outputB is null while inputB is not.

Implements

ITruncator