Class ByteFallbackDecoder

Converts tokens looking like "<0x61>" to character, and attempts to concatenate them into a string. If the tokens cannot be decoded, '�' is used instead for each inconvertible byte token.

Inheritance

object

ByteFallbackDecoder

Implements

IDecoder

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Unity.InferenceEngine.Tokenization.Decoders

Assembly: Unity.InferenceEngine.Tokenization.dll

Syntax

public class ByteFallbackDecoder : IDecoder

Methods

Decode(IReadOnlyList<string>, Output<string>)

Applies modifications to the input detokenized strings.

Declaration

public void Decode(IReadOnlyList<string> tokens, Output<string> output)

Parameters

Type	Name	Description
IReadOnlyList<string>	tokens	The string values to modify.
Output<string>	output	The recipient of modified strings.

Implements

IDecoder