Enum OperationType

Namespace: AiDotNet.Enums

Assembly: AiDotNet.dll

Represents different operation types in computation graphs for JIT compilation and automatic differentiation.

public enum OperationType

Fields

Abs = 8: Element-wise absolute value - |x| for each element.
Activation = 35: Generic activation function application.
AdaptivePooling = 146: Adaptive pooling.
Add = 2: Element-wise addition of two tensors.
AffineGrid = 64: Affine grid generation for spatial transformers.
And = 168: Logical AND.
AnomalyScore = 107: Anomaly score computation.
Attention = 130: Generic attention mechanism operation.
AveragePooling = 143: Average pooling operation.
AvgPool2D = 60: 2D average pooling.
BatchNorm = 62: Batch normalization.
BatchNormalization = 138: Batch normalization.
BentIdentity = 32: Bent Identity - (sqrt(x² + 1) - 1) / 2 + x, smooth alternative to ReLU.
Broadcast = 129: Broadcast operation - expands tensor dimensions to match target shape.
CELU = 30: Continuously Differentiable ELU - max(0, x) + min(0, α * (exp(x/α) - 1)).
CRFForward = 106: CRF forward algorithm for sequence labeling.
Cast = 172: Type cast operation.
CliffordInnerProduct = 80: Inner (contraction) product of multivectors.
Clip = 173: Clip values to range.
ComplexMatMul = 72: Complex matrix multiplication for quantum operations.
ComplexMultiply = 73: Element-wise complex multiplication.
Concat = 44: Concatenate multiple tensors along an axis.
Constant = 1: Constant node - represents a constant value that doesn't require gradients.
Conv2D = 52: 2D convolution operation.
ConvTranspose2D = 53: 2D transposed convolution (deconvolution).
Convolution = 132: General convolution operation.
Convolution2D = 133: 2D convolution operation.
Convolution3D = 134: 3D convolution operation.
Crop = 46: Crop tensor by removing border elements.
CrossAttention = 152: Cross-attention operation.
Custom = 126: Custom user-defined operation for extensibility.
Deconvolution = 137: Deconvolution (transposed convolution) operation.
DeformableConv2D = 57: 2D deformable convolution with learnable offsets and optional modulation.
Dense = 147: Dense (fully connected) layer.
DepthwiseConv2D = 55: 2D depthwise convolution.
DepthwiseConvolution = 135: Depthwise convolution operation.
DilatedConv2D = 54: 2D dilated (atrous) convolution.
DilatedConvolution = 136: Dilated convolution operation.
Divide = 5: Element-wise division of two tensors.
DropPath = 160: DropPath regularization.
Dropout = 127: Dropout regularization operation - randomly zeros elements during training.
ELU = 20: Exponential Linear Unit - ELU(x) = x if x > 0, alpha * (exp(x) - 1) otherwise.
Embedding = 67: Embedding lookup operation.
Equal = 163: Element-wise equality.
Exp = 9: Element-wise exponential function - e^x for each element.
Expand = 159: Expand tensor dimensions.
FakeQuantization = 125: Fake quantization operation with Straight-Through Estimator (STE) for differentiable quantization. Forward: quantized = round(x / scale) * scale Backward: gradient passes through unchanged (STE)
Flatten = 156: Flatten tensor to 1D.
FullyConnected = 148: Fully connected layer.
FusedAddReLU = 100: Fused addition + ReLU.
FusedConvBatchNorm = 99: Fused convolution + batch normalization.
FusedConvBatchNormReLU = 175: Fused Conv + BatchNorm + ReLU.
FusedLayerNormAttention = 180: Fused LayerNorm + Attention.
FusedLinearReLU = 98: Fused linear layer with ReLU (MatMul + Add + ReLU).
FusedMatMulAdd = 97: Fused matrix multiplication + addition (MatMul + Add).
FusedMatMulBias = 176: Fused MatMul + Bias.
FusedMatMulBiasGELU = 178: Fused MatMul + Bias + GELU.
FusedMatMulBiasReLU = 177: Fused MatMul + Bias + ReLU.
FusedMultiHeadAttention = 179: Fused MultiHead Attention.
GELU = 22: Gaussian Error Linear Unit - x * Φ(x) where Φ is standard normal CDF.
GRU = 154: GRU recurrent layer.
GRUCell = 70: GRU cell operation for recurrent networks.
Gather = 128: Gather operation - selects elements from a tensor using indices.
Gaussian = 33: Gaussian activation - exp(-x²), bell-shaped response curve.
Gemm = 149: General Matrix Multiplication.
GeometricProduct = 78: Geometric product of multivectors.
GlobalAveragePooling = 144: Global average pooling.
GlobalMaxPooling = 145: Global max pooling.
GradeProject = 83: Grade projection of multivectors.
GraphConv = 66: Graph convolutional operation for GNNs.
Greater = 164: Element-wise greater than.
GreaterOrEqual = 166: Element-wise greater or equal.
GridSample = 65: Grid sampling for spatial transformers.
GroupNormalization = 141: Group normalization.
GumbelSoftmax = 101: Gumbel-Softmax for differentiable discrete sampling (used in stochastic layers).
HardSigmoid = 27: Hard Sigmoid - piecewise linear approximation of sigmoid: clip((x + 1) / 2, 0, 1).
HardTanh = 28: Hard Tanh - piecewise linear approximation of tanh: clip(x, -1, 1).
HierarchicalSoftmax = 121: Hierarchical Softmax - tree-based efficient softmax for large vocabularies.
HyperboloidDistance = 88: Hyperboloid distance metric.
ISRU = 110: Inverse Square Root Unit - x / sqrt(1 + alpha * x²).
Input = 0: Input node - represents a variable or parameter in the computation graph.
InstanceNormalization = 140: Instance normalization.
LSTM = 153: LSTM recurrent layer.
LSTMCell = 71: LSTM cell operation for recurrent networks.
LayerNorm = 61: Layer normalization.
LayerNormalization = 139: Layer normalization.
LeakyReLU = 21: Leaky Rectified Linear Unit - max(alpha * x, x) where alpha is typically 0.01.
LeakyStateUpdate = 105: Leaky state update for reservoir/echo state networks.
Less = 165: Element-wise less than.
LessOrEqual = 167: Element-wise less or equal.
LiSHT = 31: Linearly Scaled Hyperbolic Tangent - x * tanh(x).
LocallyConnectedConv2D = 56: 2D locally connected convolution.
Log = 10: Element-wise natural logarithm.
LogSoftmax = 112: Log-Softmax - log(softmax(x)), numerically stable for cross-entropy loss.
LogSoftmin = 114: Log-Softmin - log(softmin(x)) = log(softmax(-x)).
MatMul = 14: Matrix multiplication (not element-wise).
MaxPool2D = 58: 2D max pooling.
MaxPool3D = 59: 3D max pooling.
MaxPooling = 142: Max pooling operation.
Maxout = 116: Maxout activation - maximum over multiple linear pieces.
Mean = 41: Mean operation (reduces all dimensions).
Mish = 24: Mish activation - x * tanh(softplus(x)).
MobiusAdd = 84: Mobius addition in Poincare ball model.
MultiHeadAttention = 69: Multi-head attention operation.
Multiply = 4: Element-wise multiplication (Hadamard product) of two tensors.
MultivectorAdd = 81: Multivector addition.
MultivectorReverse = 82: Multivector reverse operation.
Negate = 7: Element-wise negation - multiplies each element by -1.
Norm = 13: L2 norm computation along an axis - sqrt(sum(x²)).
Not = 170: Logical NOT.
OctonionAdd = 77: Octonion addition.
OctonionConjugate = 76: Octonion conjugation.
OctonionMatMul = 75: Octonion matrix multiplication for neural networks.
OctonionMultiply = 74: Octonion multiplication (non-associative).
Or = 169: Logical OR.
Output = 131: Output node in computation graph.
PReLU = 108: Parametric Rectified Linear Unit - max(0, x) + alpha * min(0, x) where alpha is learned.
Pad = 45: Pad tensor with values.
Permute = 43: Permute tensor dimensions (general transpose).
PixelShuffle = 51: Pixel shuffle operation for upsampling.
PoincareDistance = 87: Poincare ball distance metric.
PoincareExpMap = 85: Poincare exponential map.
PoincareLogMap = 86: Poincare logarithmic map.
PositionalEncoding = 161: Positional encoding for transformers.
Power = 6: Element-wise power operation - raises each element to a specified exponent.
RBFKernel = 63: RBF (Radial Basis Function) kernel operation.
RNN = 155: Basic RNN layer.
RReLU = 117: Randomized Leaky ReLU - LeakyReLU with random alpha during training.
ReLU = 16: Rectified Linear Unit - max(0, x).
ReduceLogVariance = 40: Log-variance reduction along specified axes.
ReduceMax = 39: Maximum value reduction along specified axes.
ReduceMean = 38: Mean reduction along specified axes.
ReduceMin = 150: Minimum value reduction.
ReduceSum = 37: Sum reduction along specified axes.
Reshape = 42: Reshape tensor to new dimensions.
SELU = 26: Scaled Exponential Linear Unit - self-normalizing activation with fixed lambda and alpha.
SQRBF = 115: Square Radial Basis Function - smooth bell-shaped activation.
ScaledDotProductAttention = 68: Scaled dot-product attention.
ScaledTanh = 34: Scaled Tanh - parameterized tanh with adjustable steepness β.
Scatter = 174: Scatter values to indices.
Se3Exp = 95: SE(3) exponential map.
Se3Log = 96: SE(3) logarithmic map.
SelfAttention = 151: Self-attention operation.
Sigmoid = 17: Sigmoid activation - 1 / (1 + e^(-x)).
Sign = 111: Sign function with surrogate gradient for training - returns -1, 0, or 1.
Slice = 48: Slice tensor along an axis - extract a portion with optional stride.
So3Exp = 93: SO(3) exponential map.
So3Log = 94: SO(3) logarithmic map.
SoftKNN = 123: Soft K-Nearest Neighbors operation for differentiable instance-based learning. Uses attention-weighted contributions from all support vectors instead of hard k-selection. weights = softmax(-distances / temperature), output = Σ weights * labels
SoftLocallyWeighted = 124: Soft locally-weighted regression operation for differentiable instance-based learning. Uses attention-weighted linear combination of training targets based on distance. weights = softmax(-||x - X_train||² / bandwidth), output = weights @ y_train
SoftPlus = 25: SoftPlus activation - ln(1 + e^x), smooth approximation of ReLU.
SoftSign = 29: SoftSign activation - x / (1 + |x|), alternative to tanh with polynomial tails.
SoftSplit = 122: Soft split operation for differentiable decision trees. Uses sigmoid gating: p_left = σ((threshold - x[feature]) / temperature) output = p_left * left_value + (1 - p_left) * right_value
Softmax = 19: Softmax activation - converts logits to probability distribution.
Softmin = 113: Softmin - softmax(-x), assigns higher probability to lower values.
SpMM = 90: Sparse matrix-matrix multiplication.
SpMV = 89: Sparse matrix-vector multiplication.
SparseGather = 91: Sparse gather operation.
SparseScatter = 92: Sparse scatter operation.
Sparsemax = 120: Sparsemax - projects onto probability simplex, can produce sparse outputs.
SphericalSoftmax = 118: Spherical Softmax - L2 normalization followed by softmax.
Split = 47: Split tensor along an axis into multiple tensors.
Sqrt = 11: Element-wise square root.
Square = 12: Element-wise square - x² for each element.
Squash = 36: Squashing activation for capsule networks - s(v) = ||v||² / (1 + ||v||²) * (v / ||v||).
Squeeze = 157: Remove dimensions of size 1.
Stack = 162: Stack tensors along new axis.
StraightThroughThreshold = 103: Straight-through threshold for HTM-style sparse activations.
Subtract = 3: Element-wise subtraction of two tensors.
SurrogateSpike = 102: Surrogate spike function for spiking neural networks with gradient estimation.
Swish = 23: Swish/SiLU activation - x * sigmoid(x).
Tanh = 18: Hyperbolic tangent activation.
TaylorSoftmax = 119: Taylor Softmax - softmax using Taylor series approximation of exp.
ThresholdedReLU = 109: Thresholded Rectified Linear Unit - x if x > threshold, 0 otherwise.
TopKSoftmax = 104: Top-K softmax for mixture-of-experts routing.
Transpose = 15: Matrix transpose - swaps rows and columns.
Unknown = 181: Unknown operation type.
Unsqueeze = 158: Add dimension of size 1.
Upsample = 49: Upsample tensor by repeating elements.
Upsample3D = 50: 3D upsampling operation for volumetric data. Increases spatial resolution by repeating or interpolating values in depth, height, and width.
WedgeProduct = 79: Wedge (outer) product of multivectors.
Xor = 171: Logical XOR.

Remarks

For Beginners: Operation types identify mathematical operations performed on tensors in neural networks.

When building a computation graph, each operation (like adding two tensors or applying an activation function) needs to be identified so that:

The JIT compiler can optimize the code
The automatic differentiation system can compute gradients correctly
The system can analyze and transform the computation graph

This enum provides type-safe identification of operations, preventing typos and enabling better tooling support.

Table of Contents

Enum OperationType

Fields

Remarks