Table of Contents

Interface ICommunicationBackend<T>

Namespace
AiDotNet.DistributedTraining
Assembly
AiDotNet.dll

Defines the contract for distributed communication backends.

public interface ICommunicationBackend<T>

Type Parameters

T

The numeric type for operations (float, double, etc.)

Remarks

This abstraction allows different implementations (in-memory, MPI.NET, NCCL, etc.) to provide collective communication operations for distributed training.

For Beginners: This interface defines how different processes (or GPUs) communicate with each other during distributed training. Think of it as a "walkie-talkie" system where multiple processes can send data to each other, synchronize, and perform collective operations.

Properties

IsInitialized

Gets whether this backend is initialized and ready for use.

bool IsInitialized { get; }

Property Value

bool

Rank

Gets the rank (ID) of the current process in the distributed group.

int Rank { get; }

Property Value

int

Remarks

Rank 0 is typically the "master" or "coordinator" process.

For Beginners: Think of rank as your process's unique ID number. If you have 4 GPUs, ranks will be 0, 1, 2, and 3. Rank 0 is usually the "boss" that coordinates everything.

WorldSize

Gets the total number of processes in the distributed group.

int WorldSize { get; }

Property Value

int

Remarks

For Beginners: This is how many processes (or GPUs) are working together. If WorldSize is 4, you have 4 processes sharing the work.

Methods

AllGather(Vector<T>)

AllGather operation - gathers data from all processes and concatenates it.

Vector<T> AllGather(Vector<T> sendData)

Parameters

sendData Vector<T>

The local data to contribute

Returns

Vector<T>

The gathered data from all processes concatenated together

Remarks

Each process receives the complete concatenated result.

For Beginners: If GPU 0 has [1,2], GPU 1 has [3,4], GPU 2 has [5,6], GPU 3 has [7,8], then AllGather gives everyone [1,2,3,4,5,6,7,8]. This is used to reconstruct the full model parameters from sharded pieces.

AllReduce(Vector<T>, ReductionOperation)

AllReduce operation - combines data from all processes using the specified operation and distributes the result back to all processes.

void AllReduce(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce. Will be replaced with the reduced result.

operation ReductionOperation

The reduction operation (Sum, Max, Min, etc.)

Remarks

For Beginners: Imagine 4 GPUs each calculated a gradient vector. AllReduce takes all 4 vectors, adds them together (if operation is Sum), and gives the result to all 4 GPUs. This is crucial for averaging gradients across GPUs during training.

Common operations: - Sum: Add all values together (used for gradient averaging) - Max: Take the maximum value across all processes - Min: Take the minimum value across all processes

Barrier()

Synchronization barrier - blocks until all processes reach this point.

void Barrier()

Remarks

For Beginners: This is like a meeting checkpoint. All processes must arrive at this point before any of them can continue. It ensures everyone is synchronized. Example: Before starting training, you want all GPUs to be ready.

Broadcast(Vector<T>, int)

Broadcast operation - sends data from one process (root) to all other processes.

Vector<T> Broadcast(Vector<T> data, int root = 0)

Parameters

data Vector<T>

The data to broadcast (only meaningful on root process)

root int

The rank of the process that is broadcasting

Returns

Vector<T>

The broadcast data (received from root on non-root processes)

Remarks

For Beginners: This is like an announcement from the boss (root process). The root sends data to everyone else. Useful for distributing initial parameters or configurations.

Initialize()

Initializes the communication backend.

void Initialize()

Remarks

Must be called before any other operations.

For Beginners: This is like turning on your walkie-talkie system. You need to do this once at the start before any processes can talk to each other.

Receive(int, int, int)

Receive operation - receives data from a specific source process.

Vector<T> Receive(int sourceRank, int count, int tag = 0)

Parameters

sourceRank int

The rank of the process to receive from

count int

The expected number of elements to receive

tag int

Optional message tag to match with Send (default=0)

Returns

Vector<T>

The received data

Remarks

This is a point-to-point communication operation that blocks until data arrives.

For Beginners: This is like waiting for a private message from a specific GPU. The process will wait (block) until the message arrives.

Use cases:

  • Pipeline parallelism: receiving activations from previous stage
  • Ring-based algorithms: receiving data from neighbor
  • Custom communication patterns

Important: Receive must be matched with a corresponding Send from the source process. If the sender never sends, this will deadlock (hang forever). If the sizes don't match, data corruption or errors can occur.

ReduceScatter(Vector<T>, ReductionOperation)

ReduceScatter operation - reduces data and scatters the result.

Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce and scatter

operation ReductionOperation

The reduction operation

Returns

Vector<T>

The reduced chunk for this process

Remarks

Combines AllReduce and Scatter in one operation for efficiency.

For Beginners: This is an optimization that combines reduction and scattering. Instead of doing AllReduce (everyone gets everything) then Scatter (split it up), we directly compute and distribute only the needed chunks.

Scatter(Vector<T>, int)

Scatter operation - distributes different chunks of data from root to each process.

Vector<T> Scatter(Vector<T> sendData, int root = 0)

Parameters

sendData Vector<T>

The data to scatter (only used on root process)

root int

The rank of the process that is scattering

Returns

Vector<T>

The chunk of data received by this process

Remarks

For Beginners: The root has a big array and wants to give each GPU a different piece. If root has [1,2,3,4,5,6,7,8] and WorldSize=4, it gives: GPU 0 gets [1,2], GPU 1 gets [3,4], GPU 2 gets [5,6], GPU 3 gets [7,8]

Send(Vector<T>, int, int)

Send operation - sends data from this process to a specific destination process.

void Send(Vector<T> data, int destinationRank, int tag = 0)

Parameters

data Vector<T>

The data to send

destinationRank int

The rank of the process to send to

tag int

Optional message tag to distinguish different messages (default=0)

Remarks

This is a point-to-point communication operation. Unlike collective operations (AllReduce, Broadcast, etc.), only two processes are involved: sender and receiver.

For Beginners: This is like sending a private message to one specific GPU. Unlike Broadcast (which sends to everyone), Send only sends to one receiver.

Use cases:

  • Pipeline parallelism: sending activations from one stage to the next
  • Ring-based algorithms: sending data to neighbor in a ring
  • Custom communication patterns

Important: Send must be matched with a corresponding Receive on the destination process. The sender and receiver must agree on the message size, otherwise deadlock or incorrect data transfer can occur.

Shutdown()

Shuts down the communication backend and releases resources. Should be called when distributed training is complete.

void Shutdown()
Edit this page