Class GlooCommunicationBackend<T>

Namespace

AiDotNet.DistributedTraining

Assembly

AiDotNet.dll

Gloo-based communication backend for CPU-based collective operations.

public class GlooCommunicationBackend<T> : CommunicationBackendBase<T>, ICommunicationBackend<T>

Type Parameters

T

The numeric type for operations

Inheritance

object

CommunicationBackendBase<T>

GlooCommunicationBackend<T>

Implements

ICommunicationBackend<T>

Inherited Members

CommunicationBackendBase<T>.NumOps

CommunicationBackendBase<T>.IsInitialized

CommunicationBackendBase<T>.Initialize()

CommunicationBackendBase<T>.Shutdown()

CommunicationBackendBase<T>.EnsureInitialized()

CommunicationBackendBase<T>.ValidateRoot(int)

CommunicationBackendBase<T>.ValidateRank(int, string)

CommunicationBackendBase<T>.ValidateData(Vector<T>, string)

CommunicationBackendBase<T>.ApplyReductionOperation(T, T, ReductionOperation)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Overview: Gloo is Facebook's collective communications library optimized for both CPUs and GPUs. It provides efficient implementations of collective operations for CPU-based training or heterogeneous environments. Gloo is particularly well-suited for training on CPUs or mixed CPU/GPU clusters where NCCL may not be available or optimal.

Features: - CPU-optimized collective operations - Supports TCP, InfiniBand via ibverbs - Works on both CPUs and GPUs - Cross-platform (Linux, macOS, Windows) - Used by PyTorch's distributed package

Use Cases: - CPU-based distributed training - Heterogeneous clusters (mixed CPU/GPU) - When NCCL is not available (non-NVIDIA hardware, macOS, etc.) - Development and testing on laptops/workstations - Production training on CPU clusters

Requirements: - Gloo library (C++) - .NET bindings for Gloo (custom P/Invoke or wrapper library) - Network connectivity between workers (TCP/IP or InfiniBand)

Architecture: This backend supports two modes of operation:

1. Native Gloo Mode (Optional): Requires GlooSharp package (separate NuGet) which provides .NET bindings for the native Gloo C++ library. Gloo offers optimized collective operations for CPU and GPU. To use: Install the GlooSharp package separately.

2. Built-in TCP Mode (Default, Production-Ready): Production-ready TCP-based implementation using industry-standard ring algorithms (ring-allreduce, ring-allgather, ring-reduce-scatter). Provides full multi-process functionality without external dependencies.

The TCP implementation features:

• Automatic TCP connection initialization with retry logic and handshakes
• Ring-based collective operations for optimal bandwidth utilization
• Proper error handling, validation, and timeout mechanisms
• Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
• Support for arbitrary world sizes and fault-tolerant connection establishment

Recommendation: Use TCP mode for most scenarios. Add GlooSharp only if you need specialized hardware support (InfiniBand) or have specific Gloo optimizations.

Constructors

GlooCommunicationBackend(int, int)

Creates a new Gloo communication backend using production-ready TCP implementation.

public GlooCommunicationBackend(int rank = 0, int worldSize = 1)

Parameters

rank int

This process's rank

worldSize int

Total number of processes

Remarks

This backend uses TCP-based collective operations via ring algorithms. For native Gloo library support with InfiniBand, see GitHub issue #461.

Properties

Rank

Gets the rank (ID) of the current process in the distributed group.

public override int Rank { get; }

Property Value

int

Remarks

Rank 0 is typically the "master" or "coordinator" process.

For Beginners: Think of rank as your process's unique ID number. If you have 4 GPUs, ranks will be 0, 1, 2, and 3. Rank 0 is usually the "boss" that coordinates everything.

WorldSize

Gets the total number of processes in the distributed group.

public override int WorldSize { get; }

Property Value

int

Remarks

For Beginners: This is how many processes (or GPUs) are working together. If WorldSize is 4, you have 4 processes sharing the work.

Methods

AllGather(Vector<T>)

AllGather operation - gathers data from all processes and concatenates it.

public override Vector<T> AllGather(Vector<T> sendData)

Parameters

sendData Vector<T>

The local data to contribute

Returns

Vector<T>

The gathered data from all processes concatenated together

Remarks

Each process receives the complete concatenated result.

For Beginners: If GPU 0 has [1,2], GPU 1 has [3,4], GPU 2 has [5,6], GPU 3 has [7,8], then AllGather gives everyone [1,2,3,4,5,6,7,8]. This is used to reconstruct the full model parameters from sharded pieces.

AllReduce(Vector<T>, ReductionOperation)

AllReduce operation - combines data from all processes using the specified operation and distributes the result back to all processes.

public override void AllReduce(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce. Will be replaced with the reduced result.

operation ReductionOperation

The reduction operation (Sum, Max, Min, etc.)

Remarks

For Beginners: Imagine 4 GPUs each calculated a gradient vector. AllReduce takes all 4 vectors, adds them together (if operation is Sum), and gives the result to all 4 GPUs. This is crucial for averaging gradients across GPUs during training.

Common operations: - Sum: Add all values together (used for gradient averaging) - Max: Take the maximum value across all processes - Min: Take the minimum value across all processes

Barrier()

Synchronization barrier - blocks until all processes reach this point.

public override void Barrier()

Remarks

For Beginners: This is like a meeting checkpoint. All processes must arrive at this point before any of them can continue. It ensures everyone is synchronized. Example: Before starting training, you want all GPUs to be ready.

Broadcast(Vector<T>, int)

Broadcast operation - sends data from one process (root) to all other processes.

public override Vector<T> Broadcast(Vector<T> data, int root = 0)

Parameters

data Vector<T>

The data to broadcast (only meaningful on root process)

root int

The rank of the process that is broadcasting

Returns

Vector<T>

The broadcast data (received from root on non-root processes)

Remarks

For Beginners: This is like an announcement from the boss (root process). The root sends data to everyone else. Useful for distributing initial parameters or configurations.

OnInitialize()

Called during initialization to perform backend-specific setup.

protected override void OnInitialize()

Remarks

Derived classes override this method to implement their specific initialization logic, such as connecting to MPI or setting up shared memory structures.

For Beginners: This is where each specific backend does its setup work.

For example:

• An MPI backend would connect to the MPI environment
• An in-memory backend would create shared data structures
• An NCCL backend would initialize GPU communication channels

OnShutdown()

Called during shutdown to perform backend-specific cleanup.

protected override void OnShutdown()

Remarks

Derived classes override this method to implement their specific cleanup logic, such as disconnecting from MPI or releasing shared memory.

For Beginners: This is where each backend cleans up its resources.

It's like turning off equipment when you're done - releasing memory, closing connections, and ensuring everything shuts down cleanly.

Receive(int, int, int)

Receive operation - receives data from a specific source process.

public override Vector<T> Receive(int sourceRank, int count, int tag = 0)

Parameters

sourceRank int

The rank of the process to receive from

count int

The expected number of elements to receive

tag int

Optional message tag to match with Send (default=0)

Returns

Vector<T>

The received data

Remarks

This is a point-to-point communication operation that blocks until data arrives.

For Beginners: This is like waiting for a private message from a specific GPU. The process will wait (block) until the message arrives.

Use cases:

• Pipeline parallelism: receiving activations from previous stage
• Ring-based algorithms: receiving data from neighbor
• Custom communication patterns

Important: Receive must be matched with a corresponding Send from the source process. If the sender never sends, this will deadlock (hang forever). If the sizes don't match, data corruption or errors can occur.

ReduceScatter(Vector<T>, ReductionOperation)

ReduceScatter operation - reduces data and scatters the result.

public override Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)

Parameters

data Vector<T>

The data to reduce and scatter

operation ReductionOperation

The reduction operation

Returns

Vector<T>

The reduced chunk for this process

Remarks

Combines AllReduce and Scatter in one operation for efficiency.

For Beginners: This is an optimization that combines reduction and scattering. Instead of doing AllReduce (everyone gets everything) then Scatter (split it up), we directly compute and distribute only the needed chunks.

Scatter(Vector<T>, int)

Scatter operation - distributes different chunks of data from root to each process.

public override Vector<T> Scatter(Vector<T> sendData, int root = 0)

Parameters

sendData Vector<T>

The data to scatter (only used on root process)

root int

The rank of the process that is scattering

Returns

Vector<T>

The chunk of data received by this process

Remarks

For Beginners: The root has a big array and wants to give each GPU a different piece. If root has [1,2,3,4,5,6,7,8] and WorldSize=4, it gives: GPU 0 gets [1,2], GPU 1 gets [3,4], GPU 2 gets [5,6], GPU 3 gets [7,8]

Send(Vector<T>, int, int)

Send operation - sends data from this process to a specific destination process.

public override void Send(Vector<T> data, int destinationRank, int tag = 0)

Parameters

data Vector<T>

The data to send

destinationRank int

The rank of the process to send to

tag int

Optional message tag to distinguish different messages (default=0)

Remarks

This is a point-to-point communication operation. Unlike collective operations (AllReduce, Broadcast, etc.), only two processes are involved: sender and receiver.

For Beginners: This is like sending a private message to one specific GPU. Unlike Broadcast (which sends to everyone), Send only sends to one receiver.

Use cases:

• Pipeline parallelism: sending activations from one stage to the next
• Ring-based algorithms: sending data to neighbor in a ring
• Custom communication patterns

Important: Send must be matched with a corresponding Receive on the destination process. The sender and receiver must agree on the message size, otherwise deadlock or incorrect data transfer can occur.