Table of Contents

Class OneCycleLRScheduler

Namespace
AiDotNet.LearningRateSchedulers
Assembly
AiDotNet.dll

Implements the 1cycle learning rate policy.

public class OneCycleLRScheduler : LearningRateSchedulerBase, ILearningRateScheduler
Inheritance
OneCycleLRScheduler
Implements
Inherited Members

Examples

// 1cycle policy over 100 epochs with peak LR of 0.1
var scheduler = new OneCycleLRScheduler(
    maxLearningRate: 0.1,
    totalSteps: 100,
    pctStart: 0.3,      // 30% warmup
    divFactor: 25,      // Start LR = 0.1/25 = 0.004
    finalDivFactor: 1e4 // End LR = 0.1/10000 = 0.00001
);

Remarks

The 1cycle policy starts with a low learning rate, increases it to a maximum, then decreases it again. This approach has been shown to enable faster training and better final performance, especially when combined with momentum cycling.

For Beginners: The 1cycle policy is like warming up before a workout, going full intensity during the workout, and then cooling down. The learning rate starts low (warmup), ramps up to a maximum (peak training), and then decreases to very low values (fine-tuning). This approach often allows training with higher maximum learning rates and can achieve better results in fewer epochs.

Based on the paper "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates" by Leslie N. Smith and Nicholay Topin.

Constructors

OneCycleLRScheduler(double, int, double, double, double, AnnealingStrategy)

Initializes a new instance of the OneCycleLRScheduler class.

public OneCycleLRScheduler(double maxLearningRate, int totalSteps, double pctStart = 0.3, double divFactor = 25, double finalDivFactor = 10000, OneCycleLRScheduler.AnnealingStrategy annealStrategy = AnnealingStrategy.Cosine)

Parameters

maxLearningRate double

The maximum learning rate (peak of the cycle).

totalSteps int

Total number of steps (typically epochs * steps_per_epoch).

pctStart double

Percentage of the cycle spent increasing the learning rate. Default: 0.3

divFactor double

Factor to determine initial learning rate (initial_lr = max_lr / div_factor). Default: 25

finalDivFactor double

Factor to determine final learning rate (final_lr = initial_lr / final_div_factor). Default: 10000

annealStrategy OneCycleLRScheduler.AnnealingStrategy

Annealing strategy for the decay phase. Default: Cosine

Exceptions

ArgumentException

Thrown when parameters are invalid.

Properties

MaxLearningRate

Gets the maximum learning rate.

public double MaxLearningRate { get; }

Property Value

double

PctStart

Gets the percentage of steps for warmup.

public double PctStart { get; }

Property Value

double

TotalSteps

Gets the total number of steps.

public int TotalSteps { get; }

Property Value

int

Methods

ComputeLearningRate(int)

Computes the learning rate for a given step.

protected override double ComputeLearningRate(int step)

Parameters

step int

The step number.

Returns

double

The computed learning rate.

GetState()

Gets the scheduler state for serialization/checkpointing.

public override Dictionary<string, object> GetState()

Returns

Dictionary<string, object>

A dictionary containing the scheduler state.

Edit this page