Table of Contents

Class MiniBatchGradientDescentOptimizer<T, TInput, TOutput>

Namespace
AiDotNet.Optimizers
Assembly
AiDotNet.dll

Implements the Mini-Batch Gradient Descent optimization algorithm.

public class MiniBatchGradientDescentOptimizer<T, TInput, TOutput> : GradientBasedOptimizerBase<T, TInput, TOutput>, IGradientBasedOptimizer<T, TInput, TOutput>, IOptimizer<T, TInput, TOutput>, IModelSerializer

Type Parameters

T

The numeric type used for calculations, typically float or double.

TInput
TOutput
Inheritance
OptimizerBase<T, TInput, TOutput>
GradientBasedOptimizerBase<T, TInput, TOutput>
MiniBatchGradientDescentOptimizer<T, TInput, TOutput>
Implements
IGradientBasedOptimizer<T, TInput, TOutput>
IOptimizer<T, TInput, TOutput>
Inherited Members
Extension Methods

Remarks

Mini-Batch Gradient Descent is a variation of gradient descent that splits the training data into small batches to calculate model error and update model coefficients. This approach strikes a balance between the efficiency of stochastic gradient descent and the stability of batch gradient descent.

For Beginners: Imagine you're trying to find the bottom of a valley while blindfolded. Mini-Batch Gradient Descent is like taking a few steps, checking your position, adjusting your direction, and repeating. It's faster than checking after every single step (Stochastic Gradient Descent) but more precise than taking a lot of steps before checking (Batch Gradient Descent).

Constructors

MiniBatchGradientDescentOptimizer(IFullModel<T, TInput, TOutput>, MiniBatchGradientDescentOptions<T, TInput, TOutput>?, IEngine?)

Initializes a new instance of the MiniBatchGradientDescentOptimizer class.

public MiniBatchGradientDescentOptimizer(IFullModel<T, TInput, TOutput> model, MiniBatchGradientDescentOptions<T, TInput, TOutput>? options = null, IEngine? engine = null)

Parameters

model IFullModel<T, TInput, TOutput>

The model to optimize.

options MiniBatchGradientDescentOptions<T, TInput, TOutput>
engine IEngine

Remarks

This constructor sets up the optimizer with the provided options and dependencies. If no options are provided, it uses default settings. It also initializes a random number generator for shuffling data.

For Beginners: This is like setting up your hiking gear before starting the journey to find the valley's bottom. You're deciding on your strategy (options) and packing your tools (dependencies) that you'll use along the way.

Methods

Deserialize(byte[])

Deserializes a byte array to restore the optimizer's state.

public override void Deserialize(byte[] data)

Parameters

data byte[]

The byte array containing the serialized optimizer state.

Remarks

This method takes a byte array (previously created by Serialize) and uses it to restore the optimizer's state, including its base class state and options.

For Beginners: This is like using a detailed map and instructions to recreate your exact position and plan from a previous point in your journey. It allows you to pick up right where you left off, with all your strategies and progress intact.

Exceptions

InvalidOperationException

Thrown when the optimizer options cannot be deserialized.

GenerateGradientCacheKey(IFullModel<T, TInput, TOutput>, TInput, TOutput)

Generates a unique key for caching gradients based on the model and input data.

protected override string GenerateGradientCacheKey(IFullModel<T, TInput, TOutput> model, TInput X, TOutput y)

Parameters

model IFullModel<T, TInput, TOutput>

The symbolic model being optimized.

X TInput

The input data matrix.

y TOutput

The target output vector.

Returns

string

A string representing the unique gradient cache key.

Remarks

This method creates a unique identifier for caching gradients. It combines the base gradient cache key with specific parameters of the Mini-Batch Gradient Descent algorithm.

For Beginners: Imagine you're leaving markers along your hiking path. This method creates a unique label for each marker, combining information about where you are (the model and data) with specifics about how you're hiking (batch size and number of rounds). This helps you quickly recognize and use information from similar situations you've encountered before.

GetOptions()

Gets the current optimization algorithm options.

public override OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()

Returns

OptimizationAlgorithmOptions<T, TInput, TOutput>

The current MiniBatchGradientDescentOptions object.

Remarks

This method returns the current options used by the Mini-Batch Gradient Descent optimizer.

For Beginners: This is like checking your current hiking plan. It lets you see all the settings and strategies you're currently using in your journey to find the valley's bottom.

InitializeAdaptiveParameters()

Initializes adaptive parameters for the optimization process.

protected override void InitializeAdaptiveParameters()

Remarks

This method sets up the initial learning rate for the optimization process based on the options provided.

For Beginners: This is like deciding how big your steps will be when you start your journey. The learning rate determines how much you adjust your position based on each batch of information you process.

Optimize(OptimizationInputData<T, TInput, TOutput>)

Performs the optimization process using Mini-Batch Gradient Descent.

public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)

Parameters

inputData OptimizationInputData<T, TInput, TOutput>

The input data for the optimization process.

Returns

OptimizationResult<T, TInput, TOutput>

The result of the optimization process.

Remarks

This method implements the main optimization loop. It iterates through the data in mini-batches, calculating gradients and updating the model parameters for each batch. The process continues for a specified number of epochs or until a stopping criterion is met.

For Beginners: This is the actual journey to find the valley's bottom. You're taking steps (processing batches of data), checking your position (evaluating the model), and adjusting your direction (updating the model parameters). You do this repeatedly (for each epoch) until you're satisfied with your position or you've taken the maximum number of steps you allowed yourself.

DataLoader Integration: This optimizer now uses the DataLoader batching infrastructure which supports: - Custom samplers (weighted, stratified, curriculum, importance, active learning) - Reproducible shuffling via RandomSeed - Option to drop incomplete final batches Set these options via GradientBasedOptimizerOptions.DataSampler, ShuffleData, DropLastBatch, and RandomSeed.

ReverseUpdate(Vector<T>, Vector<T>)

Reverses a Mini-Batch Gradient Descent update to recover original parameters.

public override Vector<T> ReverseUpdate(Vector<T> updatedParameters, Vector<T> appliedGradients)

Parameters

updatedParameters Vector<T>

Parameters after Mini-Batch GD update

appliedGradients Vector<T>

The gradients that were applied

Returns

Vector<T>

Original parameters before the update

Remarks

Mini-Batch Gradient Descent uses vanilla SGD update rule: params_new = params_old - lr * gradient. The reverse is straightforward: params_old = params_new + lr * gradient.

For Beginners: This calculates where parameters were before a Mini-Batch GD update. Since Mini-Batch GD uses simple steps (parameter minus learning_rate times gradient), reversing just means adding back that step.

Serialize()

Serializes the optimizer's state into a byte array.

public override byte[] Serialize()

Returns

byte[]

A byte array representing the serialized state of the optimizer.

Remarks

This method converts the current state of the optimizer, including its base class state and options, into a byte array. This is useful for saving the optimizer's state or transferring it between systems.

For Beginners: Think of this as taking a snapshot of your entire journey so far. It captures all the details of your current position, your hiking plan, and how you got there. This snapshot can be used to continue your journey later or share your exact situation with others.

UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput>, OptimizationStepData<T, TInput, TOutput>)

Updates the adaptive parameters of the optimizer based on the current and previous optimization steps.

protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput> currentStepData, OptimizationStepData<T, TInput, TOutput> previousStepData)

Parameters

currentStepData OptimizationStepData<T, TInput, TOutput>

Data from the current optimization step.

previousStepData OptimizationStepData<T, TInput, TOutput>

Data from the previous optimization step.

Remarks

This method adjusts the learning rate based on the performance of the current step compared to the previous step. If improvement is seen, the learning rate may be increased, otherwise it may be decreased.

For Beginners: This is like adjusting your step size based on how well you're doing. If you're making good progress, you might take slightly bigger steps. If you're not improving, you might take smaller, more careful steps.

UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput>)

Updates the optimizer's options with new settings.

protected override void UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput> options)

Parameters

options OptimizationAlgorithmOptions<T, TInput, TOutput>

The new options to be applied to the optimizer.

Remarks

This method allows updating the optimizer's settings during runtime. It ensures that only compatible option types are used with this optimizer.

For Beginners: This is like changing your hiking strategy mid-journey. It makes sure you're only using strategies that work for this specific type of journey (Mini-Batch Gradient Descent).

Exceptions

ArgumentException

Thrown when the provided options are not of the correct type.

UpdateParametersGpu(IGpuBuffer, IGpuBuffer, int, IDirectGpuBackend)

Updates parameters on the GPU using vanilla SGD (same as SGD for parameter updates).

public override void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend)

Parameters

parameters IGpuBuffer
gradients IGpuBuffer
parameterCount int
backend IDirectGpuBackend

UpdateSolution(IFullModel<T, TInput, TOutput>, Vector<T>)

Updates the current solution based on the calculated gradient.

protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)

Parameters

currentSolution IFullModel<T, TInput, TOutput>

The current model solution.

gradient Vector<T>

The calculated gradient.

Returns

IFullModel<T, TInput, TOutput>

An updated symbolic model with improved coefficients.

Remarks

This method applies the gradient to the current solution, adjusting each coefficient by the gradient scaled by the learning rate.

For Beginners: This is like taking a step in the direction you think will lead you closer to the valley's bottom. The size of your step is determined by the learning rate, and the direction is given by the gradient.

Edit this page