Enum SEALAdaptiveLearningRateMode

Namespace

AiDotNet.MetaLearning.Options

Assembly

AiDotNet.dll

Specifies the mode for computing adaptive learning rates in SEAL.

public enum SEALAdaptiveLearningRateMode

Fields

GradientNorm = 0

Computes adaptive learning rate based on instantaneous gradient norm. lr = base_lr / (sqrt(grad^2) + epsilon) Similar to AdaGrad's approach.

PerLayer = 2

Computes one adaptive rate per layer (averaged across layer parameters). Reduces noise in adaptive rate estimates for small layers.

RunningMean = 1

Uses exponential moving average of squared gradients. Similar to RMSprop's approach, providing more stable estimates.

Remarks

For Beginners: Different ways to adjust the step size for each parameter: - GradientNorm: Big gradients get smaller steps (like AdaGrad) - RunningMean: Uses a moving average of past gradients (like RMSprop) - PerLayer: All parameters in a layer share the same adaptive rate