Class NCCLCommunicationBackend<T>

Namespace

AiDotNet.DistributedTraining

Assembly

AiDotNet.dll

NVIDIA NCCL-based communication backend for GPU-to-GPU communication.

public class NCCLCommunicationBackend<T> : CommunicationBackendBase<T>, ICommunicationBackend<T>

Type Parameters

T

The numeric type for operations

Inheritance

object

CommunicationBackendBase<T>

NCCLCommunicationBackend<T>

Implements

ICommunicationBackend<T>

Inherited Members

CommunicationBackendBase<T>.NumOps

CommunicationBackendBase<T>.IsInitialized

CommunicationBackendBase<T>.Initialize()

CommunicationBackendBase<T>.Shutdown()

CommunicationBackendBase<T>.EnsureInitialized()

CommunicationBackendBase<T>.ValidateRoot(int)

CommunicationBackendBase<T>.ValidateRank(int, string)

CommunicationBackendBase<T>.ValidateData(Vector<T>, string)

CommunicationBackendBase<T>.ApplyReductionOperation(T, T, ReductionOperation)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Overview: NCCL (NVIDIA Collective Communications Library) is optimized for multi-GPU communication on NVIDIA GPUs. It provides highly optimized implementations of collective operations that take advantage of NVLink, PCIe, and network topology for maximum throughput.

Features: - Optimized for NVIDIA GPUs (NVLink, NVSwitch awareness) - Near-optimal bandwidth utilization - Supports multi-node multi-GPU configurations - Ring and tree algorithms for different collective operations - Essential for high-performance multi-GPU training

Architecture: This backend supports two modes of operation:

1. Native NCCL Mode: Uses NCCL library with actual GPU memory for collective operations. Requires CUDA toolkit and NCCL library. Provides near-optimal GPU bandwidth.

2. CPU Fallback Mode: When NCCL/CUDA not available, uses TCP-based ring algorithms similar to Gloo. Allows development and testing on systems without NVIDIA GPUs.

The implementation features:

• Automatic NCCL detection and initialization
• TCP-based unique ID distribution for multi-node setup
• Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
• Proper CUDA stream synchronization
• Memory-efficient GPU operations

Requirements for GPU Mode: - NVIDIA GPUs (compute capability 3.0+) - CUDA toolkit 10.0+ - NCCL library 2.0+

Constructors

NCCLCommunicationBackend(int, int, int)

Creates a new NCCL communication backend.

public NCCLCommunicationBackend(int rank = 0, int worldSize = 1, int deviceId = -1)

Parameters

rank int

This process's rank

worldSize int

Total number of processes

deviceId int

CUDA device ID for this process (default: use rank)

Properties

Rank

Gets the rank (ID) of the current process in the distributed group.

public override int Rank { get; }

Property Value

int

Remarks

Rank 0 is typically the "master" or "coordinator" process.

For Beginners: Think of rank as your process's unique ID number. If you have 4 GPUs, ranks will be 0, 1, 2, and 3. Rank 0 is usually the "boss" that coordinates everything.

WorldSize

Gets the total number of processes in the distributed group.

public override int WorldSize { get; }

Property Value

int

Remarks

For Beginners: This is how many processes (or GPUs) are working together. If WorldSize is 4, you have 4 processes sharing the work.

Methods

AllGather(Vector<T>)

AllGather operation - gathers data from all processes and concatenates it.

public override Vector<T> AllGather(Vector<T> sendData)

Parameters

sendData Vector<T>

The local data to contribute

Returns

Vector<T>

The gathered data from all processes concatenated together

Remarks

Each process receives the complete concatenated result.

For Beginners: If GPU 0 has [1,2], GPU 1 has [3,4], GPU 2 has [5,6], GPU 3 has [7,8], then AllGather gives everyone [1,2,3,4,5,6,7,8]. This is used to reconstruct the full model parameters from sharded pieces.

AllReduce(Vector<T>, ReductionOperation)

AllReduce operation - combines data from all processes using the specified operation and distributes the result back to all processes.

public override void AllReduce(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce. Will be replaced with the reduced result.

operation ReductionOperation

The reduction operation (Sum, Max, Min, etc.)

Remarks

For Beginners: Imagine 4 GPUs each calculated a gradient vector. AllReduce takes all 4 vectors, adds them together (if operation is Sum), and gives the result to all 4 GPUs. This is crucial for averaging gradients across GPUs during training.

Common operations: - Sum: Add all values together (used for gradient averaging) - Max: Take the maximum value across all processes - Min: Take the minimum value across all processes

Barrier()

Synchronization barrier - blocks until all processes reach this point.

public override void Barrier()

Remarks

For Beginners: This is like a meeting checkpoint. All processes must arrive at this point before any of them can continue. It ensures everyone is synchronized. Example: Before starting training, you want all GPUs to be ready.

Broadcast(Vector<T>, int)

Broadcast operation - sends data from one process (root) to all other processes.

public override Vector<T> Broadcast(Vector<T> data, int root = 0)

Parameters

data Vector<T>

The data to broadcast (only meaningful on root process)

root int

The rank of the process that is broadcasting

Returns

Vector<T>

The broadcast data (received from root on non-root processes)

Remarks

For Beginners: This is like an announcement from the boss (root process). The root sends data to everyone else. Useful for distributing initial parameters or configurations.

OnInitialize()

Called during initialization to perform backend-specific setup.

protected override void OnInitialize()

Remarks

Derived classes override this method to implement their specific initialization logic, such as connecting to MPI or setting up shared memory structures.

For Beginners: This is where each specific backend does its setup work.

For example:

• An MPI backend would connect to the MPI environment
• An in-memory backend would create shared data structures
• An NCCL backend would initialize GPU communication channels

OnShutdown()

Called during shutdown to perform backend-specific cleanup.

protected override void OnShutdown()

Remarks

Derived classes override this method to implement their specific cleanup logic, such as disconnecting from MPI or releasing shared memory.

For Beginners: This is where each backend cleans up its resources.

It's like turning off equipment when you're done - releasing memory, closing connections, and ensuring everything shuts down cleanly.

Receive(int, int, int)

Receive operation - receives data from a specific source process.

public override Vector<T> Receive(int sourceRank, int count, int tag = 0)

Parameters

sourceRank int

The rank of the process to receive from

count int

The expected number of elements to receive

tag int

Optional message tag to match with Send (default=0)

Returns

Vector<T>

The received data

Remarks

This is a point-to-point communication operation that blocks until data arrives.

For Beginners: This is like waiting for a private message from a specific GPU. The process will wait (block) until the message arrives.

Use cases:

• Pipeline parallelism: receiving activations from previous stage
• Ring-based algorithms: receiving data from neighbor
• Custom communication patterns

Important: Receive must be matched with a corresponding Send from the source process. If the sender never sends, this will deadlock (hang forever). If the sizes don't match, data corruption or errors can occur.

ReduceScatter(Vector<T>, ReductionOperation)

ReduceScatter operation - reduces data and scatters the result.

public override Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce and scatter

operation ReductionOperation

The reduction operation

Returns

Vector<T>

The reduced chunk for this process

Remarks

Combines AllReduce and Scatter in one operation for efficiency.

For Beginners: This is an optimization that combines reduction and scattering. Instead of doing AllReduce (everyone gets everything) then Scatter (split it up), we directly compute and distribute only the needed chunks.

Scatter(Vector<T>, int)

Scatter operation - distributes different chunks of data from root to each process.

public override Vector<T> Scatter(Vector<T> sendData, int root = 0)

Parameters

sendData Vector<T>

The data to scatter (only used on root process)

root int

The rank of the process that is scattering

Returns

Vector<T>

The chunk of data received by this process

Remarks

For Beginners: The root has a big array and wants to give each GPU a different piece. If root has [1,2,3,4,5,6,7,8] and WorldSize=4, it gives: GPU 0 gets [1,2], GPU 1 gets [3,4], GPU 2 gets [5,6], GPU 3 gets [7,8]

Send(Vector<T>, int, int)

Send operation - sends data from this process to a specific destination process.

public override void Send(Vector<T> data, int destinationRank, int tag = 0)

Parameters

data Vector<T>

The data to send

destinationRank int

The rank of the process to send to

tag int

Optional message tag to distinguish different messages (default=0)

Remarks

This is a point-to-point communication operation. Unlike collective operations (AllReduce, Broadcast, etc.), only two processes are involved: sender and receiver.

For Beginners: This is like sending a private message to one specific GPU. Unlike Broadcast (which sends to everyone), Send only sends to one receiver.

Use cases:

• Pipeline parallelism: sending activations from one stage to the next
• Ring-based algorithms: sending data to neighbor in a ring
• Custom communication patterns

Important: Send must be matched with a corresponding Receive on the destination process. The sender and receiver must agree on the message size, otherwise deadlock or incorrect data transfer can occur.