Table of Contents

Class DenseLoRAAdapter<T>

Namespace
AiDotNet.LoRA.Adapters
Assembly
AiDotNet.dll

LoRA adapter specifically for Dense and FullyConnected layers with 1D input/output shapes.

public class DenseLoRAAdapter<T> : LoRAAdapterBase<T>, IDisposable, ILoRAAdapter<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
DenseLoRAAdapter<T>
Implements
Inherited Members

Remarks

The DenseLoRAAdapter wraps Dense or FullyConnected layers and adds a LoRA layer in parallel. During forward pass, both the base layer and LoRA layer process the input, and their outputs are summed. The base layer's parameters can be frozen while only the LoRA parameters are trained.

For Beginners: This adapter lets you add LoRA to Dense or FullyConnected layers. Think of it like adding a "correction layer" that learns what adjustments are needed:

  • The base layer keeps its original weights (optionally frozen)
  • The LoRA layer learns a small correction
  • The final output is: original_output + lora_correction

This is incredibly useful for fine-tuning pre-trained models:

  1. Load a pre-trained model with Dense/FullyConnected layers
  2. Wrap those layers with DenseLoRAAdapter
  3. Freeze the base layers
  4. Train only the small LoRA corrections
  5. Achieve similar results with 100x fewer trainable parameters!

Example: If you have a dense layer with 1000x1000 weights, wrapping it with rank=8 LoRA (frozen) reduces trainable parameters from 1,000,000 to just 16,000!

Constructors

DenseLoRAAdapter(ILayer<T>, int, double, bool)

Initializes a new Dense LoRA adapter wrapping an existing Dense or FullyConnected layer.

public DenseLoRAAdapter(ILayer<T> baseLayer, int rank, double alpha = -1, bool freezeBaseLayer = true)

Parameters

baseLayer ILayer<T>

The Dense or FullyConnected layer to adapt with LoRA.

rank int

The rank of the LoRA decomposition.

alpha double

The LoRA scaling factor (defaults to rank if negative).

freezeBaseLayer bool

Whether to freeze the base layer's parameters during training.

Remarks

For Beginners: This creates an adapter that adds LoRA to a Dense or FullyConnected layer.

Parameters:

  • baseLayer: The Dense or FullyConnected layer you want to make more efficient to fine-tune
  • rank: How much compression (lower = fewer parameters, less flexibility)
  • alpha: How strong the LoRA adaptation is
  • freezeBaseLayer: Whether to lock the original layer's weights (usually true for efficiency)

This adapter only works with layers that have 1D input/output shapes, which includes:

  • DenseLayer (standard fully connected layer)
  • FullyConnectedLayer (another name for the same thing)

It validates that the base layer has compatible shapes before proceeding.

Exceptions

ArgumentNullException

Thrown when baseLayer is null.

ArgumentException

Thrown when the base layer doesn't have 1D input/output shapes.

Methods

MergeToOriginalLayer()

Merges the LoRA adaptation into the base layer and returns the merged Dense layer.

public override ILayer<T> MergeToOriginalLayer()

Returns

ILayer<T>

A new DenseLayer with LoRA weights merged into the base layer's weights.

Remarks

This method supports merging for both DenseLayer and FullyConnectedLayer base layers. The LoRA weights are computed and added directly to the base layer's weight matrix.

For Beginners: This "bakes in" your LoRA adaptation to create a regular Dense layer. After training with LoRA, you can merge the adaptation into the original weights for: - Faster inference (no need to compute LoRA separately) - Simpler deployment (single layer instead of two) - Compatibility with systems that don't support LoRA

Think of it like merging tracked changes in a document - you go from "original + changes" to a single updated version.

The merging process:

  1. Gets the LoRA weight matrix (computed from A and B matrices)
  2. Adds these weights to the base layer's existing weights
  3. Copies biases unchanged (LoRA doesn't modify biases)
  4. Creates a new DenseLayer with the merged weights

Exceptions

InvalidOperationException

Thrown when the base layer type is not DenseLayer or FullyConnectedLayer.

Edit this page