Table of Contents

Class CyclicLRScheduler

Namespace
AiDotNet.LearningRateSchedulers
Assembly
AiDotNet.dll

Implements cyclical learning rate policy.

public class CyclicLRScheduler : LearningRateSchedulerBase, ILearningRateScheduler
Inheritance
CyclicLRScheduler
Implements
Inherited Members

Examples

// Triangular mode cycling between 0.001 and 0.1
var scheduler = new CyclicLRScheduler(
    baseLearningRate: 0.001,
    maxLearningRate: 0.1,
    stepSizeUp: 2000,
    mode: CyclicLRScheduler.CyclicMode.Triangular
);

Remarks

CyclicLR cycles the learning rate between two boundaries with a constant frequency. This approach can help escape local minima and find better solutions by periodically increasing the learning rate.

For Beginners: Instead of always decreasing the learning rate, cyclic learning rates go up and down in cycles. The idea is that periodically increasing the learning rate can help the model escape local minima (suboptimal solutions) and explore better solutions. Think of it like occasionally taking bigger jumps while hiking to avoid getting stuck in small valleys.

Based on the paper "Cyclical Learning Rates for Training Neural Networks" by Leslie N. Smith.

Constructors

CyclicLRScheduler(double, double, int, int?, CyclicMode, double)

Initializes a new instance of the CyclicLRScheduler class.

public CyclicLRScheduler(double baseLearningRate, double maxLearningRate, int stepSizeUp = 2000, int? stepSizeDown = null, CyclicLRScheduler.CyclicMode mode = CyclicMode.Triangular, double gamma = 1)

Parameters

baseLearningRate double

Minimum learning rate.

maxLearningRate double

Maximum learning rate.

stepSizeUp int

Number of training iterations in the increasing half of a cycle. Default: 2000

stepSizeDown int?

Number of training iterations in the decreasing half. Default: same as stepSizeUp

mode CyclicLRScheduler.CyclicMode

Cycling mode. Default: Triangular

gamma double

Constant for 'exp_range' mode, scales amplitude by gamma^cycle. Default: 1.0

Properties

CycleCount

Gets the current cycle count.

public int CycleCount { get; }

Property Value

int

MaxLearningRate

Gets the maximum learning rate.

public double MaxLearningRate { get; }

Property Value

double

StepSizeDown

Gets the step size for decreasing phase.

public int StepSizeDown { get; }

Property Value

int

StepSizeUp

Gets the step size for increasing phase.

public int StepSizeUp { get; }

Property Value

int

Methods

ComputeLearningRate(int)

Computes the learning rate for a given step.

protected override double ComputeLearningRate(int step)

Parameters

step int

The step number.

Returns

double

The computed learning rate.

GetState()

Gets the scheduler state for serialization/checkpointing.

public override Dictionary<string, object> GetState()

Returns

Dictionary<string, object>

A dictionary containing the scheduler state.

Edit this page