Class DistributedExtensions

Namespace

AiDotNet.DistributedTraining

Assembly

AiDotNet.dll

Provides extension methods for easily enabling distributed training on models and optimizers.

public static class DistributedExtensions

Inheritance

object

DistributedExtensions

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

For Beginners: Extension methods allow you to add new functionality to existing classes without modifying them. These extensions add a simple .AsDistributed() method to models and optimizers, making it incredibly easy to enable distributed training with just one line of code!

Example:

// Original model
var model = new NeuralNetworkModel<double>(...);
// Make it distributed with one line!
var distributedModel = model.AsDistributed(backend);
// That's it! Now train as usual
distributedModel.Train(inputs, outputs);

Methods

AsDistributedForHighBandwidth<T, TInput, TOutput>(IFullModel<T, TInput, TOutput>, ICommunicationBackend<T>)

Creates a distributed model with optimized settings for high-bandwidth networks.

public static IShardedModel<T, TInput, TOutput> AsDistributedForHighBandwidth<T, TInput, TOutput>(this IFullModel<T, TInput, TOutput> model, ICommunicationBackend<T> communicationBackend)

Parameters

model IFullModel<T, TInput, TOutput>

The model to make distributed

communicationBackend ICommunicationBackend<T>

The communication backend to use

Returns

IShardedModel<T, TInput, TOutput>

A distributed model optimized for high-bandwidth networks

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: Use this when your GPUs are connected with fast connections (like NVLink). It automatically configures settings optimized for high-speed networks.

AsDistributedForLowBandwidth<T, TInput, TOutput>(IFullModel<T, TInput, TOutput>, ICommunicationBackend<T>)

Creates a distributed model with optimized settings for low-bandwidth networks.

public static IShardedModel<T, TInput, TOutput> AsDistributedForLowBandwidth<T, TInput, TOutput>(this IFullModel<T, TInput, TOutput> model, ICommunicationBackend<T> communicationBackend)

Parameters

model IFullModel<T, TInput, TOutput>

The model to make distributed

communicationBackend ICommunicationBackend<T>

The communication backend to use

Returns

IShardedModel<T, TInput, TOutput>

A distributed model optimized for low-bandwidth networks

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: Use this when your machines are connected over slower networks (like ethernet). It automatically configures settings to minimize network traffic.

AsDistributed<T, TInput, TOutput>(IFullModel<T, TInput, TOutput>, ICommunicationBackend<T>)

Wraps a model with distributed training capabilities using the specified communication backend.

public static IShardedModel<T, TInput, TOutput> AsDistributed<T, TInput, TOutput>(this IFullModel<T, TInput, TOutput> model, ICommunicationBackend<T> communicationBackend)

Parameters

model IFullModel<T, TInput, TOutput>

The model to make distributed

communicationBackend ICommunicationBackend<T>

The communication backend to use

Returns

IShardedModel<T, TInput, TOutput>

A distributed version of the model

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: This is the "magic" method that makes any model distributed. Just call: myModel.AsDistributed(backend)

What it does: 1. Creates a configuration for distributed training 2. Wraps your model in a ShardedModel 3. Returns the distributed version of your model

The distributed model works exactly like the original, but automatically: - Shards parameters across processes - Synchronizes gradients during training - Coordinates with other processes

Exceptions

ArgumentNullException

Thrown if model or backend is null

AsDistributed<T, TInput, TOutput>(IFullModel<T, TInput, TOutput>, IShardingConfiguration<T>)

Wraps a model with distributed training capabilities using a custom configuration.

public static IShardedModel<T, TInput, TOutput> AsDistributed<T, TInput, TOutput>(this IFullModel<T, TInput, TOutput> model, IShardingConfiguration<T> configuration)

Parameters

model IFullModel<T, TInput, TOutput>

The model to make distributed

configuration IShardingConfiguration<T>

The sharding configuration to use

Returns

IShardedModel<T, TInput, TOutput>

A distributed version of the model

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: This is the same as the basic AsDistributed(), but lets you customize settings. Use this when you need more control over how distributed training works.

Example:

var config = new ShardingConfiguration<double>(backend)
{
    AutoSyncGradients = true,
    MinimumParameterGroupSize = 2048,
    EnableGradientCompression = true
};
var distributedModel = myModel.AsDistributed(config);

Exceptions

ArgumentNullException

Thrown if model or configuration is null

AsDistributed<T, TInput, TOutput>(IOptimizer<T, TInput, TOutput>, ICommunicationBackend<T>)

Wraps an optimizer with distributed training capabilities.

public static IShardedOptimizer<T, TInput, TOutput> AsDistributed<T, TInput, TOutput>(this IOptimizer<T, TInput, TOutput> optimizer, ICommunicationBackend<T> communicationBackend)

Parameters

optimizer IOptimizer<T, TInput, TOutput>

The optimizer to make distributed

communicationBackend ICommunicationBackend<T>

The communication backend to use

Returns

IShardedOptimizer<T, TInput, TOutput>

A distributed version of the optimizer

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: Just like models, optimizers can be made distributed with one line: myOptimizer.AsDistributed(backend)

What it does: 1. Creates a configuration for distributed training 2. Wraps your optimizer in a ShardedOptimizer 3. Returns the distributed version

The distributed optimizer coordinates optimization across all processes, ensuring they stay synchronized.

Exceptions

ArgumentNullException

Thrown if optimizer or backend is null

AsDistributed<T, TInput, TOutput>(IOptimizer<T, TInput, TOutput>, IShardingConfiguration<T>)

Wraps an optimizer with distributed training capabilities using a custom configuration.

public static IShardedOptimizer<T, TInput, TOutput> AsDistributed<T, TInput, TOutput>(this IOptimizer<T, TInput, TOutput> optimizer, IShardingConfiguration<T> configuration)

Parameters

optimizer IOptimizer<T, TInput, TOutput>

The optimizer to make distributed

configuration IShardingConfiguration<T>

The sharding configuration to use

Returns

IShardedOptimizer<T, TInput, TOutput>

A distributed version of the optimizer

Type Parameters

T

The numeric type

TInput

The input type for the model

TOutput

The output type for the model

Remarks

For Beginners: Same as the basic AsDistributed() for optimizers, but with custom settings.

Example:

var config = ShardingConfiguration<double>.CreateForLowBandwidth(backend);
var distributedOptimizer = myOptimizer.AsDistributed(config);

Exceptions

ArgumentNullException

Thrown if optimizer or configuration is null