Class FusedMultiHeadAttentionOp

Namespace

AiDotNet.JitCompiler.IR.Operations

Assembly

AiDotNet.dll

Fused multi-head attention operation.

public class FusedMultiHeadAttentionOp : IROp

Inheritance

object

IROp

FusedMultiHeadAttentionOp

Inherited Members

IROp.OutputIds

IROp.OutputId

IROp.InputIds

IROp.OutputType

IROp.OutputShape

IROp.OutputShapes

IROp.OpType

IROp.ToString()

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Remarks

For Beginners: Multi-head attention for transformers.

Splits Q, K, V into multiple heads, applies attention, then concatenates. This is the complete attention layer including all projections.

Properties

CausalMask

Gets or sets whether to use causal masking.

public bool CausalMask { get; set; }

Property Value

bool

Dropout

Gets or sets dropout probability.

public double Dropout { get; set; }

Property Value

double

HeadDim

Gets or sets the head dimension.

public int HeadDim { get; set; }

Property Value

int

NumHeads

Gets or sets the number of attention heads.

public int NumHeads { get; set; }

Property Value

int

Methods

Validate()

Validates inputs (query, key, value).

public override bool Validate()

Returns

bool