Table of Contents

Class TrustRegionOptimizer<T, TInput, TOutput>

Namespace
AiDotNet.Optimizers
Assembly
AiDotNet.dll

Implements the Trust Region optimization algorithm for machine learning models.

public class TrustRegionOptimizer<T, TInput, TOutput> : GradientBasedOptimizerBase<T, TInput, TOutput>, IGradientBasedOptimizer<T, TInput, TOutput>, IOptimizer<T, TInput, TOutput>, IModelSerializer

Type Parameters

T

The numeric type used for calculations, typically float or double.

TInput
TOutput
Inheritance
OptimizerBase<T, TInput, TOutput>
GradientBasedOptimizerBase<T, TInput, TOutput>
TrustRegionOptimizer<T, TInput, TOutput>
Implements
IGradientBasedOptimizer<T, TInput, TOutput>
IOptimizer<T, TInput, TOutput>
Inherited Members
Extension Methods

Remarks

The Trust Region optimizer is an advanced optimization technique that uses local quadratic approximations of the objective function to determine the next step. It maintains a region of trust around the current solution where the approximation is considered reliable.

For Beginners: Think of this optimizer as an explorer with a map: - The "trust region" is like the area on the map the explorer trusts to be accurate. - In each step, the explorer looks at this trusted area to decide where to go next. - If the predictions (map) match reality well, the explorer might expand the trusted area. - If the predictions are off, the explorer shrinks the trusted area and becomes more cautious.

This approach helps the optimizer make good decisions even in complex landscapes, balancing between making progress and staying reliable.

Constructors

TrustRegionOptimizer(IFullModel<T, TInput, TOutput>, TrustRegionOptimizerOptions<T, TInput, TOutput>?, IEngine?)

Initializes a new instance of the TrustRegionOptimizer class.

public TrustRegionOptimizer(IFullModel<T, TInput, TOutput> model, TrustRegionOptimizerOptions<T, TInput, TOutput>? options = null, IEngine? engine = null)

Parameters

model IFullModel<T, TInput, TOutput>

The model to optimize.

options TrustRegionOptimizerOptions<T, TInput, TOutput>

Options for configuring the Trust Region optimizer.

engine IEngine

The computation engine (CPU or GPU) for vectorized operations.

Methods

Deserialize(byte[])

Deserializes the optimizer's state from a byte array.

public override void Deserialize(byte[] data)

Parameters

data byte[]

The byte array containing the serialized optimizer state.

Remarks

This method reconstructs the optimizer's state from a serialized byte array. It restores both the base class state and the specific properties of the Trust Region optimizer.

For Beginners: This is like reconstructing the optimizer from a snapshot: - It takes the snapshot (byte array) created by the Serialize method. - From this snapshot, it rebuilds the optimizer to the exact state it was in when serialized. - This is useful for resuming a paused optimization process or moving it to a different machine. - If there's a problem reading the optimizer's settings, it will raise an error to let you know.

Exceptions

InvalidOperationException

Thrown when deserialization of optimizer options fails.

GetOptions()

Retrieves the current options of the optimizer.

public override OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()

Returns

OptimizationAlgorithmOptions<T, TInput, TOutput>

The current TrustRegionOptimizerOptions.

Remarks

This method provides access to the current configuration of the Trust Region optimizer. It allows inspection of the optimizer's settings without modifying them.

For Beginners: This is like checking your current GPS settings: - You can see how the optimizer is currently configured. - This is useful if you want to review or log the current settings. - It doesn't change anything; it just lets you look at the current setup.

InitializeAdaptiveParameters()

Initializes adaptive parameters for the Trust Region optimizer.

protected override void InitializeAdaptiveParameters()

Optimize(OptimizationInputData<T, TInput, TOutput>)

Performs the optimization process using the Trust Region algorithm.

public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)

Parameters

inputData OptimizationInputData<T, TInput, TOutput>

The input data for optimization.

Returns

OptimizationResult<T, TInput, TOutput>

The result of the optimization process.

Remarks

DataLoader Integration: This method uses the DataLoader API for epoch management. Trust Region methods typically operate on the full dataset because they construct a local quadratic model of the objective function that requires accurate gradient and Hessian information. The method notifies the sampler of epoch starts using NotifyEpochStart(int) for compatibility with curriculum learning and sampling strategies.

Serialize()

Serializes the current state of the optimizer into a byte array.

public override byte[] Serialize()

Returns

byte[]

A byte array representing the serialized state of the optimizer.

Remarks

This method saves the current state of the optimizer, including its base class state and specific Trust Region optimizer properties. This allows the optimizer's state to be stored or transmitted and later reconstructed.

For Beginners: This is like taking a snapshot of the optimizer: - It captures all the important information about the optimizer's current state. - This snapshot can be saved or sent somewhere else. - Later, you can use this snapshot to recreate the optimizer exactly as it was. - It's useful for things like saving progress, or moving the optimization process to a different machine.

UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput>, OptimizationStepData<T, TInput, TOutput>)

Updates adaptive parameters based on the current and previous optimization steps.

protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput, TOutput> currentStepData, OptimizationStepData<T, TInput, TOutput> previousStepData)

Parameters

currentStepData OptimizationStepData<T, TInput, TOutput>

Data from the current optimization step.

previousStepData OptimizationStepData<T, TInput, TOutput>

Data from the previous optimization step.

Remarks

This method adjusts the trust region radius based on the improvement in the objective function. It's part of the adaptive behavior of the trust region algorithm, allowing it to respond to the characteristics of the objective function landscape.

For Beginners: This is like learning from your recent experiences: - If your last step improved things, you might become a bit more adventurous. - If things got worse, you'd become more cautious. - This method automatically adjusts how boldly or cautiously the algorithm explores, based on whether recent steps have been successful or not.

UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput>)

Updates the optimizer options.

protected override void UpdateOptions(OptimizationAlgorithmOptions<T, TInput, TOutput> options)

Parameters

options OptimizationAlgorithmOptions<T, TInput, TOutput>

The new options to be set.

Remarks

This method allows updating the optimizer's configuration during runtime. It ensures that only the correct type of options (TrustRegionOptimizerOptions) can be set for this optimizer.

For Beginners: This is like changing the settings on your GPS mid-journey: - You can adjust how the optimizer behaves without starting over. - It checks to make sure you're using the right kind of settings for this specific optimizer. - If you try to use the wrong type of settings, it will let you know with an error.

Exceptions

ArgumentException

Thrown when the provided options are not of type TrustRegionOptimizerOptions.

UpdateParameters(Vector<T>, Vector<T>)

Updates parameters using a simplified Cauchy-point Trust Region step.

public override Vector<T> UpdateParameters(Vector<T> parameters, Vector<T> gradient)

Parameters

parameters Vector<T>

The current parameters.

gradient Vector<T>

The gradient at the current parameters.

Returns

Vector<T>

The updated parameters.

Remarks

This implements a simplified Trust Region step based on the Cauchy point, which uses only gradient information. The step is constrained to lie within the trust region radius. The trust region radius is adapted based on the success of previous steps.

UpdateParametersGpu(IGpuBuffer, IGpuBuffer, int, IDirectGpuBackend)

Updates parameters using GPU-accelerated Trust Region method.

public override void UpdateParametersGpu(IGpuBuffer parameters, IGpuBuffer gradients, int parameterCount, IDirectGpuBackend backend)

Parameters

parameters IGpuBuffer
gradients IGpuBuffer
parameterCount int
backend IDirectGpuBackend
Edit this page