Table of Contents

Class MultiLoRAAdapter<T>

Namespace
AiDotNet.LoRA.Adapters
Assembly
AiDotNet.dll

Multi-task LoRA adapter that manages multiple task-specific LoRA layers for complex multi-task learning scenarios.

public class MultiLoRAAdapter<T> : LoRAAdapterBase<T>, IDisposable, ILoRAAdapter<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
MultiLoRAAdapter<T>
Implements
Inherited Members

Remarks

MultiLoRA extends the basic LoRA concept to handle multiple tasks simultaneously within a single layer. Instead of having one LoRA adaptation, it maintains a dictionary of task-specific LoRA layers, with a routing mechanism to select the appropriate adapter for each task.

Key features: - Multiple task-specific LoRA adapters sharing the same base layer - Dynamic task switching during inference and training - Per-task rank configuration for optimal parameter efficiency - Shared base layer weights across all tasks - Task-specific merging for deployment

For Beginners: Think of MultiLoRA as having one teacher (the base layer) and multiple students (task-specific LoRA adapters), each specializing in different subjects.

In regular LoRA:

  • You have one base layer (the teacher)
  • One LoRA adapter (one student learning one subject)
  • Output = base + lora_adaptation

In MultiLoRA:

  • You have one base layer (the teacher)
  • Multiple LoRA adapters (multiple students, each specializing in different tasks)
  • Output = base + task_specific_lora_adaptation

This is powerful for:

  1. Multi-domain learning: Train on medical, legal, and technical documents simultaneously
  2. Multi-lingual models: One adapter per language
  3. Multi-task learning: Sentiment analysis, named entity recognition, question answering, etc.
  4. Continual learning: Add new tasks without forgetting old ones

Example use case:

  • Base: Pre-trained language model
  • Task 1: Sentiment analysis (rank=4)
  • Task 2: Named entity recognition (rank=8)
  • Task 3: Question answering (rank=16)

You can switch between tasks at runtime, and each task only trains its specific LoRA weights!

Constructors

MultiLoRAAdapter(ILayer<T>, string, int, double, bool)

Initializes a new Multi-LoRA adapter with an initial default task.

public MultiLoRAAdapter(ILayer<T> baseLayer, string defaultTaskName, int defaultRank, double alpha = -1, bool freezeBaseLayer = true)

Parameters

baseLayer ILayer<T>

The layer to adapt with multiple LoRA adapters.

defaultTaskName string

The name of the default task.

defaultRank int

The rank for the default task's LoRA layer.

alpha double

The LoRA scaling factor (defaults to rank if negative).

freezeBaseLayer bool

Whether to freeze the base layer's parameters during training.

Remarks

The adapter is initialized with one default task. Additional tasks can be added using AddTask().

For Beginners: This creates a MultiLoRA adapter starting with one task. Think of it like creating a multi-tool that starts with one blade, and you can add more tools later.

Parameters:

  • baseLayer: The shared foundation layer (like the handle of a multi-tool)
  • defaultTaskName: A name for your first task (e.g., "sentiment", "translation")
  • defaultRank: How complex this task's adaptation is (higher = more parameters)
  • alpha: Strength of the adaptation
  • freezeBaseLayer: Whether to lock the base layer (usually true to save memory)

After creation, you can add more tasks with different ranks optimized for each task's complexity.

Exceptions

ArgumentNullException

Thrown when baseLayer or defaultTaskName is null.

ArgumentException

Thrown when defaultTaskName is empty or whitespace.

Properties

CurrentTask

Gets or sets the name of the currently active task.

public string CurrentTask { get; set; }

Property Value

string

Remarks

Changing this property switches which task-specific adapter is used during forward/backward passes. This allows dynamic task switching during inference or training.

For Beginners: This is like switching between different "modes" of your model. Set it to "sentiment" for sentiment analysis, "ner" for named entity recognition, etc. The base layer stays the same, but the adaptation changes based on the task.

Exceptions

ArgumentException

Thrown when trying to set a task that hasn't been added.

NumberOfTasks

Gets the number of tasks configured in this adapter.

public int NumberOfTasks { get; }

Property Value

int

ParameterCount

Gets the total parameter count across all task adapters.

public override int ParameterCount { get; }

Property Value

int

Remarks

This includes parameters from the base layer (if not frozen) plus all task-specific LoRA layers.

TaskAdapters

Gets the dictionary of task-specific LoRA adapters.

public IReadOnlyDictionary<string, LoRALayer<T>> TaskAdapters { get; }

Property Value

IReadOnlyDictionary<string, LoRALayer<T>>

Remarks

Each task has its own dedicated LoRA layer with potentially different ranks. This allows for task-specific parameter efficiency optimization.

Methods

AddTask(string, int, double)

Adds a new task with its own LoRA adapter.

public void AddTask(string taskName, int rank, double alpha = -1)

Parameters

taskName string

The name of the task (must be unique).

rank int

The rank for this task's LoRA layer.

alpha double

The LoRA scaling factor for this task (defaults to rank if negative).

Remarks

Each task can have a different rank, allowing you to optimize parameter usage based on task complexity. More complex tasks can use higher ranks, while simpler tasks can use lower ranks.

For Beginners: This adds a new "mode" to your model.

Example:

  • Task "sentiment" with rank=4: Simple classification (positive/negative/neutral)
  • Task "ner" with rank=8: More complex named entity recognition
  • Task "qa" with rank=16: Even more complex question answering

Each task gets its own small set of parameters (determined by rank) that learn task-specific adaptations, while all tasks share the same base layer knowledge.

Benefits:

  • Different ranks for different task complexities
  • No interference between tasks (each has separate parameters)
  • Can train tasks independently or simultaneously
  • Add new tasks without retraining existing ones

Exceptions

ArgumentException

Thrown when taskName is null, empty, whitespace, or already exists.

Backward(Tensor<T>)

Performs the backward pass through the current task's adapter.

public override Tensor<T> Backward(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

Gradient flowing back from the next layer.

Returns

Tensor<T>

Gradient to pass to the previous layer.

Remarks

The backward pass only updates the current task's LoRA parameters. Other tasks are unaffected. This allows task-specific learning without interference.

For Beginners: During training, this updates only the current task's parameters.

Benefits:

  • Training task A doesn't mess up task B's learning
  • Can train tasks one at a time or in batches
  • No "catastrophic forgetting" between tasks

The gradients flow through:

  1. Current task's LoRA layer (gets updated)
  2. Base layer (only updated if not frozen)
  3. Combined gradients flow back to previous layers

Forward(Tensor<T>)

Performs the forward pass using the currently active task's adapter.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

Input tensor.

Returns

Tensor<T>

Sum of base layer output and current task's LoRA output.

Remarks

The forward pass computes: output = base_layer(input) + current_task_lora(input)

For Beginners: This processes data through the model using the current task. 1. Input goes through the base layer (shared knowledge) 2. Input goes through the current task's LoRA layer (task-specific adaptation) 3. Results are added together

The magic: Different tasks produce different outputs even though they share the same base layer!

GetParameters()

Gets the current parameters as a vector.

public override Vector<T> GetParameters()

Returns

Vector<T>

Vector containing base parameters (if not frozen) and all task adapters' parameters.

GetTaskAdapter(string)

Gets the LoRA layer for a specific task.

public LoRALayer<T> GetTaskAdapter(string taskName)

Parameters

taskName string

The name of the task.

Returns

LoRALayer<T>

The LoRA layer for the specified task.

Remarks

For Beginners: This lets you access a specific task's LoRA layer directly. Useful for inspecting parameters, getting statistics, or manual manipulation.

Exceptions

ArgumentException

Thrown when the task doesn't exist.

GetTaskRank(string)

Gets the rank of a specific task's LoRA adapter.

public int GetTaskRank(string taskName)

Parameters

taskName string

The name of the task.

Returns

int

The rank of the task's LoRA layer.

Exceptions

ArgumentException

Thrown when the task doesn't exist.

MergeTaskToLayer(string)

Merges a specific task's LoRA weights into the base layer.

public ILayer<T> MergeTaskToLayer(string taskName)

Parameters

taskName string

The name of the task to merge.

Returns

ILayer<T>

A new layer with the specified task's LoRA weights merged into the base layer.

Remarks

This creates a deployment-ready layer for a specific task by merging its LoRA weights into the base layer. This is useful when you want to deploy a single-task model.

For Beginners: This "bakes in" one task's adaptations for deployment.

Use case:

  • You trained a MultiLoRA model with 5 tasks
  • For production, you only need the "sentiment" task
  • Call MergeTaskToLayer("sentiment") to create a standalone layer
  • Deploy just that layer (smaller, faster, simpler)

The merged layer has the base weights + that task's LoRA weights combined into one.

Exceptions

ArgumentException

Thrown when the task doesn't exist.

NotSupportedException

Thrown when the base layer type doesn't support merging.

MergeToOriginalLayer()

Merges the currently active task's LoRA weights into the base layer.

public override ILayer<T> MergeToOriginalLayer()

Returns

ILayer<T>

A new layer with current task's LoRA weights merged into the base layer.

Remarks

For Beginners: This is a shortcut to merge the current task without specifying its name. Equivalent to calling MergeTaskToLayer(CurrentTask).

RemoveTask(string)

Removes a task and its associated LoRA adapter.

public bool RemoveTask(string taskName)

Parameters

taskName string

The name of the task to remove.

Returns

bool

True if the task was removed, false if it didn't exist.

Remarks

You cannot remove the last task. At least one task must always be present. If removing the current task, the CurrentTask property will be set to the first remaining task.

For Beginners: This removes a task you no longer need. Like removing a tool from your multi-tool, but you must always keep at least one. If you remove the currently active task, the adapter automatically switches to another available task.

Exceptions

InvalidOperationException

Thrown when trying to remove the last remaining task.

ResetState()

Resets the internal state of all layers.

public override void ResetState()

Remarks

For Beginners: This clears the memory of the base layer and all task adapters. Use this when starting to process a completely new, unrelated batch of data.

SetCurrentTask(string)

Sets the current task for subsequent forward/backward operations.

public void SetCurrentTask(string taskName)

Parameters

taskName string

The name of the task to activate.

Remarks

For Beginners: This switches which task the model is currently working on. Call this before forward() to tell the model what kind of task it should perform.

Example usage:

adapter.SetCurrentTask("sentiment");
var sentimentOutput = adapter.Forward(input);

adapter.SetCurrentTask("ner");
var nerOutput = adapter.Forward(sameInput);

Same input, different outputs based on which task is active!

Exceptions

ArgumentException

Thrown when the task doesn't exist.

SetParameters(Vector<T>)

Sets the layer parameters from a vector.

public override void SetParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

Vector containing all parameters.

UpdateParameters(T)

Updates parameters for the current task only.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate for parameter updates.

Remarks

Only the current task's LoRA parameters are updated. Other tasks remain unchanged. The base layer is updated only if not frozen.

For Beginners: This is where learning happens for the current task. Only the active task's parameters get updated, leaving other tasks untouched. This is key to multi-task learning without interference!

UpdateParametersFromLayers()

Updates the parameter vector from the current layer states.

protected override void UpdateParametersFromLayers()
Edit this page