Table of Contents

Interface ILoRAAdapter<T>

Namespace
AiDotNet.Interfaces
Assembly
AiDotNet.dll

Interface for LoRA (Low-Rank Adaptation) adapters that wrap existing layers with parameter-efficient adaptations.

public interface ILoRAAdapter<T> : ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inherited Members

Remarks

LoRA adapters enable efficient fine-tuning of neural networks by learning low-rank decompositions of weight updates instead of modifying all weights directly. This interface defines the contract for all LoRA adapter implementations across different layer types.

For Beginners: A LoRA adapter wraps an existing layer (like a dense or convolutional layer) and adds a small "correction layer" that learns what adjustments are needed. This is much more memory-efficient than retraining all the weights in a large model.

Think of it like:

  • The base layer has the original knowledge (frozen or trainable)
  • The LoRA layer learns a small correction
  • The final output combines both: original + correction

This allows you to adapt large pre-trained models with 100x fewer trainable parameters!

Properties

Alpha

Gets the scaling factor (alpha) for the LoRA adaptation.

double Alpha { get; }

Property Value

double

Remarks

Alpha controls how strongly the LoRA adaptation affects the output. The actual LoRA contribution is scaled by alpha/rank. Common practice: alpha = rank (scaling factor of 1.0)

BaseLayer

Gets the base layer being adapted with LoRA.

ILayer<T> BaseLayer { get; }

Property Value

ILayer<T>

Remarks

This is the original layer that's being enhanced with LoRA adaptations. It may be frozen (non-trainable) during fine-tuning for maximum efficiency.

IsBaseLayerFrozen

Gets whether the base layer's parameters are frozen during training.

bool IsBaseLayerFrozen { get; }

Property Value

bool

Remarks

When true, only the LoRA parameters are trained, dramatically reducing memory requirements and training time. This is the typical use case for LoRA.

LoRALayer

Gets the LoRA layer providing the low-rank adaptation.

LoRALayer<T> LoRALayer { get; }

Property Value

LoRALayer<T>

Remarks

This layer implements the low-rank decomposition (A and B matrices) that provides the adaptation to the base layer's behavior.

Rank

Gets the rank of the low-rank decomposition.

int Rank { get; }

Property Value

int

Remarks

The rank determines how many parameters the LoRA adaptation uses. Lower rank = fewer parameters = more efficient but less flexible.

Typical values: - rank=1-4: Very efficient, minimal parameters - rank=8: Good balance (default for many applications) - rank=16-32: More flexibility, more parameters - rank=64+: Diminishing returns, approaching full fine-tuning

Methods

MergeToOriginalLayer()

Merges the LoRA weights back into the original layer for deployment.

ILayer<T> MergeToOriginalLayer()

Returns

ILayer<T>

A new layer with the LoRA adaptation baked into the weights.

Remarks

After training, you can merge the LoRA weights into the base layer to create a single layer that includes the adaptations. This: - Removes the overhead of parallel computation - Makes inference as fast as the original layer - Allows deployment without the LoRA infrastructure

For Beginners: Think of this as "baking in" your corrections. During training, you have original + correction computed separately. After merging, you have a single updated layer that includes both, making it faster to use in production.

Edit this page