Table of Contents

Class LevenbergMarquardtOptimizer<T, TInput, TOutput>

Namespace
AiDotNet.Optimizers
Assembly
AiDotNet.dll

Implements the Levenberg-Marquardt optimization algorithm for non-linear least squares problems.

public class LevenbergMarquardtOptimizer<T, TInput, TOutput> : GradientBasedOptimizerBase<T, TInput, TOutput>, IGradientBasedOptimizer<T, TInput, TOutput>, IOptimizer<T, TInput, TOutput>, IModelSerializer

Type Parameters

T

The numeric type used for calculations, typically float or double.

TInput
TOutput
Inheritance
OptimizerBase<T, TInput, TOutput>
GradientBasedOptimizerBase<T, TInput, TOutput>
LevenbergMarquardtOptimizer<T, TInput, TOutput>
Implements
IGradientBasedOptimizer<T, TInput, TOutput>
IOptimizer<T, TInput, TOutput>
Inherited Members
Extension Methods

Remarks

The Levenberg-Marquardt algorithm is a popular method for solving non-linear least squares problems. It combines the Gauss-Newton algorithm and the method of gradient descent, providing a robust solution that works well in many situations.

For Beginners: This optimizer is like a smart problem-solver that's really good at fitting curves to data points. It's especially useful when the relationship between your inputs and outputs isn't a straight line. It works by making small adjustments to its guess, getting closer to the best solution with each step.

Constructors

LevenbergMarquardtOptimizer(IFullModel<T, TInput, TOutput>, LevenbergMarquardtOptimizerOptions<T, TInput, TOutput>?, IEngine?)

Initializes a new instance of the LevenbergMarquardtOptimizer class.

public LevenbergMarquardtOptimizer(IFullModel<T, TInput, TOutput> model, LevenbergMarquardtOptimizerOptions<T, TInput, TOutput>? options = null, IEngine? engine = null)

Parameters

model IFullModel<T, TInput, TOutput>

The model to optimize.

options LevenbergMarquardtOptimizerOptions<T, TInput, TOutput>

Custom options for the Levenberg-Marquardt algorithm.

engine IEngine

The computation engine (CPU or GPU) for vectorized operations.

Remarks

This constructor sets up the optimizer with the provided options and dependencies. If no options are provided, it uses default settings.

For Beginners: This is where we set up our smart problem-solver. We can give it special instructions (options) on how to work, or let it use its default settings. We also give it tools (like modelEvaluator and fitDetector) that it needs to do its job well.

Methods

Deserialize(byte[])

Deserializes a byte array to restore the optimizer's state.

public override void Deserialize(byte[] data)

Parameters

data byte[]

The byte array containing the serialized optimizer state.

Remarks

This method takes a byte array (previously created by the Serialize method) and uses it to restore the optimizer's state, including its options and internal parameters.

For Beginners: This is like unpacking that snapshot we took earlier and setting up the optimizer exactly as it was when we saved it. It's the reverse process of serialization, allowing us to continue optimization from a previously saved state.

Exceptions

InvalidOperationException

Thrown when the optimizer options cannot be deserialized.

GetOptions()

Retrieves the current options of the optimizer.

public override OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()

Returns

OptimizationAlgorithmOptions<T, TInput, TOutput>

The current options of the Levenberg-Marquardt optimizer.

Remarks

This method provides access to the current settings of the Levenberg-Marquardt optimizer.

For Beginners: This is like checking the current settings on a machine. It lets you see how the optimizer is currently configured without changing anything.

InitializeAdaptiveParameters()

Initializes the adaptive parameters used by the optimizer.

protected override void InitializeAdaptiveParameters()

Remarks

This method sets up the initial values for the damping factor and iteration count. The damping factor is a key component of the Levenberg-Marquardt algorithm, controlling the balance between the Gauss-Newton and gradient descent approaches.

For Beginners: This is like setting up the starting point for our problem-solver. We give it an initial "cautiousness" level (damping factor) and reset its step counter. The cautiousness helps it decide whether to take big steps or small steps when trying to find the best solution.

Optimize(OptimizationInputData<T, TInput, TOutput>)

Performs the optimization process using the Levenberg-Marquardt algorithm.

public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)

Parameters

inputData OptimizationInputData<T, TInput, TOutput>

The input data for the optimization process.

Returns

OptimizationResult<T, TInput, TOutput>

The result of the optimization process.

Remarks

This method implements the main optimization loop. It iteratively improves the solution by calculating the Jacobian matrix, residuals, and updating the solution until a stopping criterion is met.

For Beginners: This is where the magic happens! Our problem-solver repeatedly tries to improve its guess. In each round: 1. It calculates how sensitive the output is to small changes (Jacobian). 2. It checks how far off its current guess is (residuals). 3. It uses this information to make a better guess. 4. It keeps doing this until it's happy with the result or runs out of attempts.

DataLoader Integration: This method uses the DataLoader API for epoch management. Levenberg-Marquardt typically operates on the full dataset because it requires computing the Jacobian matrix across all samples to construct the normal equations properly. The method notifies the sampler of epoch starts using NotifyEpochStart(int) for compatibility with curriculum learning and sampling strategies.

Serialize()

Serializes the optimizer's state into a byte array.

public override byte[] Serialize()

Returns

byte[]

A byte array representing the serialized state of the optimizer.

Remarks

This method converts the current state of the optimizer, including its options and internal parameters, into a byte array. This is useful for saving the optimizer's state or transferring it between systems.

For Beginners: This is like taking a snapshot of the optimizer's current state and packing it into a compact form. It's useful if you want to save the optimizer's progress and continue from this point later, or if you want to move the optimizer to a different computer.

UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput>, OptimizationStepData<T, TInput, TOutput>)

Updates the adaptive parameters of the optimizer based on the current and previous optimization steps.

protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput> currentStepData, OptimizationStepData<T, TInput, TOutput> previousStepData)

Parameters

currentStepData OptimizationStepData<T, TInput, TOutput>

Data from the current optimization step.

previousStepData OptimizationStepData<T, TInput, TOutput>

Data from the previous optimization step.

Remarks

This method adjusts the damping factor of the Levenberg-Marquardt algorithm based on the performance of the current step compared to the previous step. If the current step improved the solution, the damping factor is decreased to allow larger steps. If not, it's increased to take more conservative steps.

For Beginners: This is like adjusting how cautious our optimizer is. If the last guess was better, it becomes more confident and takes bigger steps. If the last guess wasn't so good, it becomes more careful and takes smaller steps. This helps the optimizer adapt to the shape of the problem it's solving.

UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput>)

Updates the optimizer's options with new settings.

protected override void UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput> options)

Parameters

options OptimizationAlgorithmOptions<T, TInput, TOutput>

The new options to be applied to the optimizer.

Remarks

This method allows updating the optimizer's settings during runtime. It ensures that only compatible option types are used with this optimizer.

For Beginners: This is like changing the settings on a machine while it's running. It makes sure that we're only using settings that work for this specific type of optimizer. If someone tries to use the wrong type of settings, it lets them know there's a problem.

Exceptions

ArgumentException

Thrown when the provided options are not of the correct type.

UpdateParametersGpu(IGpuBuffer, IGpuBuffer, int, IDirectGpuBackend)

Updates parameters using GPU-accelerated Levenberg-Marquardt.

public override void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend)

Parameters

parameters IGpuBuffer
gradients IGpuBuffer
parameterCount int
backend IDirectGpuBackend

Remarks

Levenberg-Marquardt is a blend of gradient descent and Gauss-Newton optimization. GPU implementation is not yet available due to the need for Jacobian computation and solving (J^T J + lambda I)^-1 J^T r systems.

Edit this page