Struct MergePair

Represents a mergeable pair of token values used in Byte Pair Encoding (BPE) tokenization. Each pair consists of two consecutive token strings that can be merged into a single token during the BPE encoding process. See BpeMapper.

Implements
Inherited Members
Namespace: Unity.InferenceEngine.Tokenization.Mappers
Assembly: Unity.InferenceEngine.Tokenization.dll
Syntax
public readonly struct MergePair : IEquatable<MergePair>

Constructors

MergePair(string, string)

Initializes a new instance of the MergePair struct with the specified token values.

Declaration
public MergePair(string first, string second)
Parameters
Type Name Description
string first

The first token value of the mergeable pair. Cannot be null.
string second

The second token value of the mergeable pair. Cannot be null.
Exceptions
Type Condition
ArgumentException

Thrown when first or second is null or empty.

Fields

First

The first token value of the mergeable pair.

Declaration
public readonly string First
Field Value
Type Description
string

Second

The second token value of the mergeable pair.

Declaration
public readonly string Second
Field Value
Type Description
string

Methods

Deconstruct(out string, out string)

Deconstructs this merge pair into its constituent token values.

Declaration
public void Deconstruct(out string first, out string second)
Parameters
Type Name Description
string first

When this method returns, contains the first token value.
string second

When this method returns, contains the second token value.

Equals(object)

Determines whether this merge pair is equal to the specified object.

Declaration
public override bool Equals(object obj)
Parameters
Type Name Description
object obj

The object to compare with this instance.
Returns
Type Description
bool

true if obj is a MergePair and is equal to this instance; otherwise, false.
Overrides

Equals(MergePair)

Determines whether this merge pair is equal to another merge pair.

Declaration
public bool Equals(MergePair other)
Parameters
Type Name Description
MergePair other

The merge pair to compare with this instance.
Returns
Type Description
bool

true if both the First and Second values are equal; otherwise, false.

GetHashCode()

Returns the hash code for this merge pair.

Declaration
public override int GetHashCode()
Returns
Type Description
int

A hash code based on both token values.
Overrides

ToString()

Returns a string representation of this merge pair in the format "(,)".

Declaration
public override string ToString()
Returns
Type Description
string

A string representation of the merge pair.
Overrides

Implements