Class TemplatePostProcessor

Post processor using the templating approach.

Inheritance

object

TemplatePostProcessor

Implements

IPostProcessor

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.PostProcessors

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class TemplatePostProcessor : IPostProcessor

Constructors

TemplatePostProcessor(Template, Template, IEnumerable<(string value, int id)>)

Initializes a new instance of the TemplatePostProcessor type.

Declaration

public TemplatePostProcessor(Template single, Template pair, IEnumerable<(string value, int id)> specialTokens)

Parameters

Type	Name	Description
Template	single	Template for processing of single sequence.
Template	pair	Template for processing of paired sequences.
IEnumerable<(string value, int id)>	specialTokens	Special tokens used in the templates.

Methods

GetNumAddedTokens(bool)

Determines the number of tokens that this IPostProcessor will add to the sequence of tokens.

Declaration

public int GetNumAddedTokens(bool isPair)

Parameters

Type	Name	Description
bool	isPair	Tells if we want the number of added tokens for a pair of sequences of tokens (true), of a single sequence (false).

Returns

Type	Description
int	Number of tokens that this IPostProcessor will add to the sequence of tokens

PostProcess(IReadOnlyList<IReadOnlyList<Token>>, IReadOnlyList<IReadOnlyList<Token>>, bool, Output<IEnumerable<IEnumerable<Token>>>)

Processes the sequence of tokens.

Declaration

public void PostProcess(IReadOnlyList<IReadOnlyList<Token>> sequenceA, IReadOnlyList<IReadOnlyList<Token>> sequenceB, bool addSpecialTokens, Output<IEnumerable<IEnumerable<Token>>> output)

Parameters

Type	Name	Description
IReadOnlyList<IReadOnlyList<Token>>	sequenceA	The single, or first sequence of tokens.
IReadOnlyList<IReadOnlyList<Token>>	sequenceB	The second sequence of a pair.
bool	addSpecialTokens	Tells whether adding a special tokens in the result sequences.
Output<IEnumerable<IEnumerable<Token>>>	output	The recipient of processed sequences.

Implements

IPostProcessor