Class LearningRateSchedulerFactory

Namespace

AiDotNet.LearningRateSchedulers

Assembly

AiDotNet.dll

Factory for creating learning rate schedulers with common configurations.

public static class LearningRateSchedulerFactory

Inheritance

object

LearningRateSchedulerFactory

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This factory provides convenient methods for creating pre-configured learning rate schedulers for common use cases. It simplifies scheduler creation and provides sensible defaults for various training scenarios.

For Beginners: Instead of manually configuring schedulers with many parameters, you can use this factory to create schedulers optimized for specific scenarios. For example, CreateForTransformer() creates a scheduler tuned for transformer model training, with warmup and linear decay that works well for attention-based models.

Methods

Create(LearningRateSchedulerType, double, int)

Creates a scheduler based on type enum.

public static ILearningRateScheduler Create(LearningRateSchedulerType type, double baseLearningRate, int totalSteps = 100)

Parameters

type LearningRateSchedulerType

The type of scheduler to create.

baseLearningRate double

The base learning rate.

totalSteps int

Total training steps (used by some schedulers).

Returns

ILearningRateScheduler

A learning rate scheduler of the specified type.

CreateAdaptive(double, double, int, double)

Creates a learning rate scheduler that adapts based on validation loss. Good when you don't know the optimal schedule in advance.

public static ILearningRateScheduler CreateAdaptive(double baseLearningRate = 0.1, double factor = 0.1, int patience = 10, double minLr = 1E-07)

Parameters

baseLearningRate double

Initial learning rate. Default: 0.1

factor double

Reduction factor. Default: 0.1

patience int

Epochs to wait before reducing. Default: 10

minLr double

Minimum learning rate. Default: 1e-7

Returns

ILearningRateScheduler

A configured ReduceOnPlateauScheduler.

CreateForCNN(double, int, double)

Creates a learning rate scheduler for typical CNN training. Uses StepLR with decay every 30 epochs.

public static ILearningRateScheduler CreateForCNN(double baseLearningRate = 0.1, int stepSize = 30, double gamma = 0.1)

Parameters

baseLearningRate double

Initial learning rate. Default: 0.1

stepSize int

Steps between LR reductions. Default: 30

gamma double

Reduction factor. Default: 0.1

Returns

ILearningRateScheduler

A configured StepLRScheduler.

CreateForFineTuning(double)

Creates a learning rate scheduler for fine-tuning pre-trained models. Uses constant low learning rate.

public static ILearningRateScheduler CreateForFineTuning(double baseLearningRate = 2E-05)

Parameters

baseLearningRate double

Learning rate for fine-tuning. Default: 2e-5

Returns

ILearningRateScheduler

A configured ConstantLRScheduler.

CreateForLongTraining(double, int, double)

Creates a learning rate scheduler for long training runs. Uses cosine annealing which works well for extended training.

public static ILearningRateScheduler CreateForLongTraining(double baseLearningRate = 0.1, int totalEpochs = 200, double etaMin = 1E-06)

Parameters

baseLearningRate double

Initial learning rate. Default: 0.1

totalEpochs int

Total training epochs. Default: 200

etaMin double

Minimum learning rate. Default: 1e-6

Returns

ILearningRateScheduler

A configured CosineAnnealingLRScheduler.

CreateForSuperConvergence(double, int, double)

Creates a learning rate scheduler for super-convergence training. Uses OneCycle policy for fast training with higher learning rates.

public static ILearningRateScheduler CreateForSuperConvergence(double maxLearningRate = 0.1, int totalSteps = 10000, double pctStart = 0.3)

Parameters

maxLearningRate double

Maximum learning rate. Default: 0.1

totalSteps int

Total training steps. Default: 10000

pctStart double

Percentage of warmup phase. Default: 0.3

Returns

ILearningRateScheduler

A configured OneCycleLRScheduler.

CreateForTransformer(double, int, int)

Creates a learning rate scheduler for transformer training. Uses linear warmup followed by linear decay.

public static ILearningRateScheduler CreateForTransformer(double baseLearningRate = 0.0001, int warmupSteps = 10000, int totalSteps = 100000)

Parameters

baseLearningRate double

Peak learning rate after warmup. Default: 1e-4

warmupSteps int

Number of warmup steps. Default: 10000

totalSteps int

Total training steps. Default: 100000

Returns

ILearningRateScheduler

A configured LinearWarmupScheduler.

CreateWithWarmRestarts(double, int, int, double)

Creates a learning rate scheduler with warm restarts. Good for escaping local minima in challenging optimization landscapes.

public static ILearningRateScheduler CreateWithWarmRestarts(double baseLearningRate = 0.1, int t0 = 10, int tMult = 2, double etaMin = 1E-06)

Parameters

baseLearningRate double

Initial learning rate. Default: 0.1

t0 int

Initial restart period. Default: 10

tMult int

Period multiplier after each restart. Default: 2

etaMin double

Minimum learning rate. Default: 1e-6

Returns

ILearningRateScheduler

A configured CosineAnnealingWarmRestartsScheduler.