Interface IPostProcessor

Transforms the sequences of tokens from the truncated output of IMapper and merges it into a single sequence.

Namespace: Unity.InferenceEngine.Tokenization.PostProcessors

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public interface IPostProcessor

Methods

GetNumAddedTokens(bool)

Determines the number of tokens that this IPostProcessor will add to the sequence of tokens.

Declaration

int GetNumAddedTokens(bool isPair)

Parameters

Type	Name	Description
bool	isPair	Tells if we want the number of added tokens for a pair of sequences of tokens (true), of a single sequence (false).

Returns

Type	Description
int	Number of tokens that this IPostProcessor will add to the sequence of tokens

PostProcess(IReadOnlyList<IReadOnlyList<Token>>, IReadOnlyList<IReadOnlyList<Token>>, bool, Output<IEnumerable<IEnumerable<Token>>>)

Processes the sequence of tokens.

Declaration

void PostProcess(IReadOnlyList<IReadOnlyList<Token>> sequenceA, IReadOnlyList<IReadOnlyList<Token>> sequenceB, bool addSpecialTokens, Output<IEnumerable<IEnumerable<Token>>> output)

Parameters

Type	Name	Description
IReadOnlyList<IReadOnlyList<Token>>	sequenceA	The single, or first sequence of tokens.
IReadOnlyList<IReadOnlyList<Token>>	sequenceB	The second sequence of a pair.
bool	addSpecialTokens	Tells whether adding a special tokens in the result sequences.
Output<IEnumerable<IEnumerable<Token>>>	output	The recipient of processed sequences.