Class LEOOptions<T, TInput, TOutput>

Namespace

AiDotNet.MetaLearning.Options

Assembly

AiDotNet.dll

Configuration options for Latent Embedding Optimization (LEO) algorithm.

public class LEOOptions<T, TInput, TOutput> : IMetaLearnerOptions<T>

Type Parameters

T

The numeric data type used for calculations (e.g., float, double).

TInput

The input data type (e.g., Matrix<T>, Tensor<T>).

TOutput

The output data type (e.g., Vector<T>, Tensor<T>).

Inheritance

object

LEOOptions<T, TInput, TOutput>

Implements

IMetaLearnerOptions<T>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

LEO performs meta-learning by learning a low-dimensional latent space for model parameters. Instead of adapting the full model parameters directly (like MAML), LEO:

1. Encodes support examples into a latent representation
2. Decodes the latent code into model parameters
3. Performs gradient descent in the latent space during adaptation
4. Decodes the adapted latent code to get final model parameters

For Beginners: Imagine you have a very large model with millions of parameters. Updating all of them during few-shot learning is slow and can lead to overfitting. LEO learns to "compress" these parameters into a much smaller space (like 64 numbers instead of millions). Adaptation happens in this compressed space, which is faster and more robust to overfitting.

Key Insight: Not all parameter configurations make sense for neural networks. By learning a latent space, LEO restricts adaptation to the "manifold" of sensible parameter settings, preventing bad updates.

Constructors

LEOOptions(IFullModel<T, TInput, TOutput>)

Initializes a new instance of the LEOOptions class with the required meta-model.

public LEOOptions(IFullModel<T, TInput, TOutput> metaModel)

Parameters

metaModel IFullModel<T, TInput, TOutput>

The feature encoder to be trained (required).

Exceptions

ArgumentNullException

Thrown when metaModel is null.

Properties

AdaptationSteps

Gets or sets the number of adaptation steps in latent space.

public int AdaptationSteps { get; set; }

Property Value

int

Default is 5.

CheckpointFrequency

Gets or sets how often to save checkpoints.

public int CheckpointFrequency { get; set; }

Property Value

int

DataLoader

Gets or sets the episodic data loader for sampling tasks. Default: null (tasks must be provided manually to MetaTrain).

public IEpisodicDataLoader<T, TInput, TOutput>? DataLoader { get; set; }

Property Value

IEpisodicDataLoader<T, TInput, TOutput>

DropoutRate

Gets or sets the dropout rate for the encoder and decoder.

public double DropoutRate { get; set; }

Property Value

double

Default is 0.0 (no dropout).

EmbeddingDimension

Gets or sets the dimension of the feature embedding.

public int EmbeddingDimension { get; set; }

Property Value

int

Default is 512.

EnableCheckpointing

Gets or sets whether to save model checkpoints.

public bool EnableCheckpointing { get; set; }

Property Value

bool

EntropyWeight

Gets or sets the entropy weight for regularizing the decoder.

public double EntropyWeight { get; set; }

Property Value

double

Default is 0.001.

EvaluationFrequency

Gets or sets how often to evaluate the meta-learner.

public int EvaluationFrequency { get; set; }

Property Value

int

EvaluationTasks

Gets or sets the number of tasks to use for evaluation.

public int EvaluationTasks { get; set; }

Property Value

int

GradientClipThreshold

Gets or sets the maximum gradient norm for gradient clipping.

public double? GradientClipThreshold { get; set; }

Property Value

double?

Default is 10.0.

HiddenDimension

Gets or sets the hidden dimension for encoder/decoder networks.

public int HiddenDimension { get; set; }

Property Value

int

Default is 256.

InnerLearningRate

Gets or sets the learning rate for the inner loop (latent space adaptation).

public double InnerLearningRate { get; set; }

Property Value

double

Default is 1.0 (full step in latent space).

Remarks

For Beginners: Since adaptation happens in a learned latent space, the learning rate can often be larger than in parameter space. The latent space is designed to make adaptation easier.

InnerOptimizer

Gets or sets the optimizer for inner loop updates. Default: null (uses SGD with InnerLearningRate).

public IGradientBasedOptimizer<T, TInput, TOutput>? InnerOptimizer { get; set; }

Property Value

IGradientBasedOptimizer<T, TInput, TOutput>

KLWeight

Gets or sets the KL divergence weight for the latent space regularization.

public double KLWeight { get; set; }

Property Value

double

Default is 0.01.

Remarks

For Beginners: LEO uses a variational approach where the latent space is regularized to be close to a prior distribution (usually Gaussian). This weight controls how strongly we enforce this regularization.

L2Regularization

Gets or sets the L2 regularization strength.

public double L2Regularization { get; set; }

Property Value

double

Default is 0.0 (no regularization).

LatentDimension

Gets or sets the dimensionality of the latent space.

public int LatentDimension { get; set; }

Property Value

int

Default is 64.

Remarks

For Beginners: This controls how "compressed" the parameter space is. Smaller values are more efficient but may limit what the model can represent. Typical values are 32-128.

LossFunction

Gets or sets the loss function for training. Default: null (uses cross-entropy loss internally).

public ILossFunction<T>? LossFunction { get; set; }

Property Value

ILossFunction<T>

MetaBatchSize

Gets or sets the number of tasks to sample per meta-training iteration.

public int MetaBatchSize { get; set; }

Property Value

int

Default is 4.

MetaModel

Gets or sets the meta-model (feature encoder) to be trained. This is the only required property.

public IFullModel<T, TInput, TOutput> MetaModel { get; set; }

Property Value

IFullModel<T, TInput, TOutput>

Remarks

The meta-model should be a feature encoder. LEO will learn to generate task-specific classifier parameters from support examples using this encoder.

MetaOptimizer

Gets or sets the optimizer for meta-parameter (outer loop) updates. Default: null (uses built-in Adam optimizer with OuterLearningRate).

public IGradientBasedOptimizer<T, TInput, TOutput>? MetaOptimizer { get; set; }

Property Value

IGradientBasedOptimizer<T, TInput, TOutput>

NumClasses

Gets or sets the number of output classes.

public int NumClasses { get; set; }

Property Value

int

Default is 5.

NumMetaIterations

Gets or sets the total number of meta-training iterations.

public int NumMetaIterations { get; set; }

Property Value

int

Default is 1000.

OuterLearningRate

Gets or sets the learning rate for the outer loop (meta-update).

public double OuterLearningRate { get; set; }

Property Value

double

Default is 0.001.

RandomSeed

Gets or sets the random seed for reproducibility.

public int? RandomSeed { get; set; }

Property Value

int?

ShareEncoder

Gets or sets whether to share the encoder across all classes.

public bool ShareEncoder { get; set; }

Property Value

bool

Default is true.

Remarks

For Beginners: If true, the same encoder is used to generate latent codes for all classes. If false, each class has its own encoder, which allows more flexibility but requires more parameters.

UseFirstOrder

Gets or sets whether to use first-order approximation.

public bool UseFirstOrder { get; set; }

Property Value

bool

Default is false (LEO uses second-order gradients through the decoder).

UseOrthogonalInit

Gets or sets whether to use orthogonal initialization for the decoder.

public bool UseOrthogonalInit { get; set; }

Property Value

bool

Default is true.

UseRelationEncoder

Gets or sets whether to use a relation network for encoding.

public bool UseRelationEncoder { get; set; }

Property Value

bool

Default is true.

Remarks

For Beginners: A relation network considers relationships between all support examples when generating the latent code, which can help when examples within a class are diverse.

Methods

Clone()

Creates a deep copy of the LEO options.

public IMetaLearnerOptions<T> Clone()

Returns

IMetaLearnerOptions<T>

A new LEOOptions instance with the same configuration.

IsValid()

Validates that all LEO configuration options are properly set.

public bool IsValid()

Returns

bool

True if the configuration is valid; otherwise, false.