Class WordPieceDecoder

An implementation of the WordPiece decoding algorithm.

Inheritance

object

WordPieceDecoder

Implements

IDecoder

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Decoders

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class WordPieceDecoder : IDecoder

Constructors

WordPieceDecoder(string, bool)

Initializes a new instance of the WordPieceDecoder type.

Declaration

public WordPieceDecoder(string prefix = "##", bool cleanup = true)

Parameters

Type	Name	Description
string	prefix	The prefix to use for subwords that are not a beginning-of-word.
bool	cleanup	Whether to cleanup some tokenization artifacts. Cleans spaces around some punctuation like ., ?, !, ,, ', and patterns (n't, 'm, do not, 've, 're, 's);

Methods

Decode(IReadOnlyList<string>, Output<string>)

Applies modifications to the input detokenized strings.

Declaration

public void Decode(IReadOnlyList<string> tokens, Output<string> output)

Parameters

Type	Name	Description
IReadOnlyList<string>	tokens	The string values to modify.
Output<string>	output	The recipient of modified strings.

Implements

IDecoder