Table of Contents

Class VBLoRAAdapter<T>

Namespace
AiDotNet.LoRA.Adapters
Assembly
AiDotNet.dll

Vector Bank LoRA (VB-LoRA) adapter that uses shared parameter banks for efficient multi-client deployment.

public class VBLoRAAdapter<T> : LoRAAdapterBase<T>, IDisposable, ILoRAAdapter<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
VBLoRAAdapter<T>
Implements
Inherited Members

Remarks

VB-LoRA (2024) introduces vector banks - reusable parameter stores shared across multiple LoRA adapters. Instead of each adapter having its own complete A and B matrices, VB-LoRA maintains global banks of column vectors. Each adapter selects which vectors from the banks to use via index arrays.

Key Innovation: Vector Bank Architecture

Traditional LoRA:

  • Each adapter stores full A (inputSize × rank) and B (rank × outputSize) matrices
  • Total parameters for N adapters = N × (inputSize × rank + rank × outputSize)
  • No sharing between adapters

VB-LoRA:

  • Global BankA contains pooled column vectors (inputSize × bankSize)
  • Global BankB contains pooled column vectors (bankSize × outputSize)
  • Each adapter stores only indices (which vectors to use from banks)
  • Total parameters = (inputSize × bankSize + bankSize × outputSize) + N × 2 × rank × sizeof(int)
  • Massive reduction when bankSize << N × rank

Benefits:

  1. Reduced Duplication: Similar adapters share vector bank entries
  2. Lower Communication Overhead: Multi-client systems can cache banks locally
  3. Memory Efficiency: Fewer unique parameters to store and transmit
  4. Scalability: Adding new adapters only requires index arrays, not full matrices
  5. Knowledge Sharing: Banks capture common adaptation patterns

For Beginners: Think of vector banks like a shared library of building blocks.

Traditional LoRA is like each person having their own complete toolbox:

  • Person 1: Full set of tools
  • Person 2: Full set of tools
  • Person 3: Full set of tools Result: Lots of duplicate tools

VB-LoRA is like a shared tool library:

  • Central tool bank: One of each tool type
  • Each person: List of which tools they need (indices)
  • Everyone shares the same physical tools Result: Much fewer tools needed overall

This is especially powerful when many adapters need similar adjustments (common in multi-task learning or personalization scenarios).

Example Scenario:

Suppose you're deploying personalized language models to 1000 users:

  • Traditional LoRA: Each user needs their own 16K parameter adapter (16MB total)
  • VB-LoRA: Shared 256K parameter bank + 1000 users × 128 indices each
  • Result: 84% memory reduction (256K + 128K vs 16M)

The shared bank captures common language patterns, while per-user indices select the patterns relevant to each individual.

Constructors

VBLoRAAdapter(ILayer<T>, int, int, int, int[]?, int[]?, double, bool, string?)

Initializes a new VB-LoRA adapter with specified bank sizes and indices.

public VBLoRAAdapter(ILayer<T> baseLayer, int rank, int bankSizeA, int bankSizeB, int[]? bankIndicesA = null, int[]? bankIndicesB = null, double alpha = -1, bool freezeBaseLayer = true, string? bankKey = null)

Parameters

baseLayer ILayer<T>

The layer to adapt with VB-LoRA.

rank int

The rank of the LoRA decomposition.

bankSizeA int

Size of the vector bank for matrix A (number of column vectors in the pool).

bankSizeB int

Size of the vector bank for matrix B (number of row vectors in the pool).

bankIndicesA int[]

Indices into Bank A (if null, random indices are selected).

bankIndicesB int[]

Indices into Bank B (if null, random indices are selected).

alpha double

The LoRA scaling factor (defaults to rank if negative).

freezeBaseLayer bool

Whether to freeze the base layer's parameters during training.

bankKey string

Unique identifier for the bank configuration (allows multiple independent banks).

Remarks

The constructor initializes or reuses existing vector banks based on the bankKey. If banks don't exist for this key, they are created with random initialization.

Bank Initialization Strategy:

  • BankA vectors: Gaussian random initialization (similar to standard LoRA matrix A)
  • BankB vectors: Zero initialization (so adapters start with no effect, like standard LoRA)
  • Indices: Random selection if not provided, allowing diverse initial configurations

For Beginners: This creates a VB-LoRA adapter. Key parameters:

  • rank: How many vectors this adapter selects from each bank (like standard LoRA rank)
  • bankSizeA/B: How many total vectors are in the shared banks (the size of the library)
  • bankIndicesA/B: Which specific vectors to use (can be left null for random selection)
  • bankKey: Allows creating separate banks for different purposes (like different model layers)

The bankKey is important: adapters with the same bankKey share banks, different keys use separate banks. This lets you have different bank pools for different layers or tasks.

Exceptions

ArgumentNullException

Thrown when baseLayer is null.

ArgumentException

Thrown when rank, bank sizes, or indices are invalid.

Properties

BankIndicesA

Gets the indices into Bank A used by this adapter.

public int[] BankIndicesA { get; }

Property Value

int[]

Remarks

These indices specify which column vectors from the global Bank A this adapter uses. The length equals the adapter's rank.

For Beginners: This is the list of which vectors from the shared library this adapter is currently using for the A matrix part of LoRA.

BankIndicesB

Gets the indices into Bank B used by this adapter.

public int[] BankIndicesB { get; }

Property Value

int[]

Remarks

These indices specify which row vectors from the global Bank B this adapter uses. The length equals the adapter's rank.

For Beginners: This is the list of which vectors from the shared library this adapter is currently using for the B matrix part of LoRA.

BankSizeA

Gets the size of Bank A (number of available column vectors).

public int BankSizeA { get; }

Property Value

int

BankSizeB

Gets the size of Bank B (number of available row vectors).

public int BankSizeB { get; }

Property Value

int

Methods

ClearBanks(string?)

Clears the global vector banks (useful for testing or reinitialization).

public static void ClearBanks(string? bankKey = null)

Parameters

bankKey string

The bank key to clear, or null to clear all banks.

Remarks

Warning: This will affect all VB-LoRA adapters using the specified bank(s). Use with caution, typically only in testing scenarios.

For Beginners: This erases the shared library. It's useful when you want to start fresh, but be careful - it affects all adapters using that library!

CreateLoRALayer(int, double)

Creates the LoRA layer for this adapter, customized to use vector bank indices.

protected override LoRALayer<T> CreateLoRALayer(int rank, double alpha)

Parameters

rank int

The rank of the LoRA decomposition.

alpha double

The LoRA scaling factor.

Returns

LoRALayer<T>

A LoRA layer configured to use vector banks.

Remarks

This override creates a standard LoRA layer. The matrices will be updated from bank vectors later in the constructor, after banks and indices are initialized.

For Beginners: This creates a basic LoRA layer first. The bank-based customization happens later in the constructor after the banks are set up.

Note: We don't call UpdateLoRALayerFromBanks here because this method is called from the base constructor, before VBLoRAAdapter's fields (_bankKey, _bankIndicesA, etc.) are initialized. The bank update happens in the constructor after all fields are set up.

Forward(Tensor<T>)

Performs the forward pass using bank-selected vectors.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

Input tensor.

Returns

Tensor<T>

Sum of base layer output and VB-LoRA output.

Remarks

The forward pass operates identically to standard LoRA, but the matrices A and B are composed of vectors selected from the shared banks.

For Beginners: This works exactly like regular LoRA forward pass, but the matrices being used are built from shared bank vectors. The computation is the same, but the memory footprint is much smaller when many adapters share banks.

GetBankA(string)

Gets the global vector bank A for inspection or advanced use cases.

public static Matrix<T>? GetBankA(string bankKey = "default")

Parameters

bankKey string

The bank key identifier.

Returns

Matrix<T>

A clone of the bank A matrix, or null if it doesn't exist.

Remarks

This method allows inspection of the shared bank state. It returns a clone to prevent accidental modification of the shared bank.

For Beginners: This lets you look at the shared library of vectors for matrix A. It's useful for debugging or understanding what vectors are available in the bank.

GetBankB(string)

Gets the global vector bank B for inspection or advanced use cases.

public static Matrix<T>? GetBankB(string bankKey = "default")

Parameters

bankKey string

The bank key identifier.

Returns

Matrix<T>

A clone of the bank B matrix, or null if it doesn't exist.

Remarks

This method allows inspection of the shared bank state. It returns a clone to prevent accidental modification of the shared bank.

For Beginners: This lets you look at the shared library of vectors for matrix B. It's useful for debugging or understanding what vectors are available in the bank.

MergeToOriginalLayer()

Merges the VB-LoRA adaptation into the base layer and returns the merged layer.

public override ILayer<T> MergeToOriginalLayer()

Returns

ILayer<T>

A new DenseLayer with VB-LoRA weights (from selected bank vectors) merged into the base layer's weights.

Remarks

This method extracts the adapter's effective weight matrix from the bank-selected vectors and merges it with the base layer, just like standard LoRA merging.

For Beginners: This "bakes in" the VB-LoRA adaptation by: 1. Extracting the vectors this adapter uses from the banks 2. Computing the full weight matrix from those vectors 3. Adding that matrix to the base layer's weights 4. Returning a regular layer with the merged weights

After merging, you don't need the banks anymore - you have a standalone layer with the adaptation permanently included.

UpdateParameters(T)

Updates parameters and propagates changes back to the shared banks.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate for parameter updates.

Remarks

After updating the LoRA layer's parameters, this method writes the changes back to the shared banks. This allows the banks to learn and improve over time as multiple adapters share training signal.

For Beginners: When training updates the adapter's vectors, we need to write those changes back to the shared library. This way, the shared bank "learns" from all adapters using it, making it better for everyone.

Think of it like updating a shared knowledge base - when one adapter learns something useful, that knowledge becomes available to all other adapters sharing the same banks.

Edit this page