Interface IMemoryBank<T>

Namespace

AiDotNet.SelfSupervisedLearning

Assembly

AiDotNet.dll

Defines the contract for memory banks used in contrastive learning methods.

public interface IMemoryBank<T>

Type Parameters

T

The numeric type used for computations (typically float or double).

Remarks

For Beginners: A memory bank is a queue that stores embeddings from previous batches. This provides a large pool of negative samples without needing huge batch sizes.

Why use a memory bank?

• Provides many negative samples (e.g., 65536) without large batch sizes
• More memory-efficient than SimCLR's in-batch negatives approach
• Consistent negative distribution across training

Used by: MoCo, MoCo v2 (not MoCo v3, SimCLR, BYOL, or SimSiam)

Example usage:

// Create memory bank with 65536 entries
var memoryBank = new MemoryBank<float>(65536, embeddingDim: 128);

// During training:
var negatives = memoryBank.GetAll();  // Get all negatives
memoryBank.Enqueue(currentEmbeddings);  // Add new embeddings

Properties

Capacity

Gets the maximum capacity of the memory bank (queue size).

int Capacity { get; }

Property Value

int

Remarks

Typical values: 4096-65536. MoCo uses 65536 by default.

CurrentSize

Gets the current number of stored embeddings.

int CurrentSize { get; }

Property Value

int

EmbeddingDimension

Gets the embedding dimension of stored vectors.

int EmbeddingDimension { get; }

Property Value

int

IsFull

Gets whether the memory bank is full (has reached capacity).

bool IsFull { get; }

Property Value

bool

Methods

Clear()

Clears all stored embeddings and resets the memory bank.

void Clear()

Enqueue(Tensor<T>)

Adds new embeddings to the memory bank (FIFO queue).

void Enqueue(Tensor<T> embeddings)

Parameters

embeddings Tensor<T>

The embeddings to add (batch of vectors).

Remarks

For Beginners: New embeddings are added to the end of the queue. When the queue is full, the oldest embeddings are removed (first-in, first-out).

GetAll()

Gets all stored embeddings for use as negative samples.

Tensor<T> GetAll()

Returns

Tensor<T>

A tensor containing all stored embeddings [CurrentSize, EmbeddingDimension].

Remarks

For Beginners: These embeddings serve as negative samples in contrastive loss. The more negatives you have, the harder and more informative the contrastive task becomes.

Sample(int)

Gets a random subset of stored embeddings.

Tensor<T> Sample(int count)

Parameters

count int

The number of embeddings to retrieve.

Returns

Tensor<T>

A tensor of randomly sampled embeddings [count, EmbeddingDimension].

Remarks

Useful when you want fewer negatives than the full memory bank.

UpdateWithMomentum(int[], Tensor<T>, double)

Updates embeddings by averaging with new values (for soft updates).

void UpdateWithMomentum(int[] indices, Tensor<T> newEmbeddings, double momentum)

Parameters

indices int[]

Indices of embeddings to update.

newEmbeddings Tensor<T>

New embedding values.

momentum double

Momentum for exponential moving average (0-1).

Remarks

Some memory bank variants use soft updates: new = momentum * old + (1 - momentum) * new