Class ZeRO3Optimizer<T, TInput, TOutput>

Namespace: AiDotNet.DistributedTraining

Assembly: AiDotNet.dll

Implements ZeRO Stage 3 optimizer - full sharding equivalent to FSDP.

public class ZeRO3Optimizer<T, TInput, TOutput> : FSDPOptimizer<T, TInput, TOutput>, IShardedOptimizer<T, TInput, TOutput>, IOptimizer<T, TInput, TOutput>, IModelSerializer

Type Parameters

T: The numeric type
TInput: The input type for the model
TOutput: The output type for the model

Inheritance: object

ShardedOptimizerBase<T, TInput, TOutput>

FSDPOptimizer<T, TInput, TOutput>

ZeRO3Optimizer<T, TInput, TOutput>

Implements: IShardedOptimizer<T, TInput, TOutput>

IOptimizer<T, TInput, TOutput>

IModelSerializer

Inherited Members: FSDPOptimizer<T, TInput, TOutput>.Optimize(OptimizationInputData<T, TInput, TOutput>)

FSDPOptimizer<T, TInput, TOutput>.SynchronizeOptimizerState()

FSDPOptimizer<T, TInput, TOutput>.ShouldEarlyStop()

FSDPOptimizer<T, TInput, TOutput>.GetOptions()

FSDPOptimizer<T, TInput, TOutput>.Serialize()

FSDPOptimizer<T, TInput, TOutput>.Deserialize(byte[])

FSDPOptimizer<T, TInput, TOutput>.SaveModel(string)

FSDPOptimizer<T, TInput, TOutput>.LoadModel(string)

ShardedOptimizerBase<T, TInput, TOutput>.NumOps

ShardedOptimizerBase<T, TInput, TOutput>.Config

ShardedOptimizerBase<T, TInput, TOutput>.WrappedOptimizer

ShardedOptimizerBase<T, TInput, TOutput>.WrappedOptimizerInternal

ShardedOptimizerBase<T, TInput, TOutput>.Rank

ShardedOptimizerBase<T, TInput, TOutput>.WorldSize

ShardedOptimizerBase<T, TInput, TOutput>.ShardingConfiguration

ShardedOptimizerBase<T, TInput, TOutput>.Optimize(OptimizationInputData<T, TInput, TOutput>)

ShardedOptimizerBase<T, TInput, TOutput>.SynchronizeOptimizerState()

ShardedOptimizerBase<T, TInput, TOutput>.SynchronizeParameters(IFullModel<T, TInput, TOutput>)

ShardedOptimizerBase<T, TInput, TOutput>.ShouldEarlyStop()

ShardedOptimizerBase<T, TInput, TOutput>.GetOptions()

ShardedOptimizerBase<T, TInput, TOutput>.LastComputedGradients

ShardedOptimizerBase<T, TInput, TOutput>.ApplyGradients(Vector<T>, IFullModel<T, TInput, TOutput>)

ShardedOptimizerBase<T, TInput, TOutput>.Reset()

ShardedOptimizerBase<T, TInput, TOutput>.Serialize()

ShardedOptimizerBase<T, TInput, TOutput>.Deserialize(byte[])

ShardedOptimizerBase<T, TInput, TOutput>.SaveModel(string)

ShardedOptimizerBase<T, TInput, TOutput>.LoadModel(string)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods: DistributedExtensions.AsDistributed<T, TInput, TOutput>(IOptimizer<T, TInput, TOutput>, ICommunicationBackend<T>)

DistributedExtensions.AsDistributed<T, TInput, TOutput>(IOptimizer<T, TInput, TOutput>, IShardingConfiguration<T>)

Remarks

Strategy Overview: ZeRO-3 is equivalent to FSDP optimizer - full sharding of parameters, gradients, and optimizer states. This class is an alias to FSDPOptimizer for ZeRO terminology consistency.

For Beginners: ZeRO-3 and FSDP optimizers are the same thing. Use whichever name you prefer. Everything is sharded for maximum memory efficiency.

Constructors

ZeRO3Optimizer(IOptimizer<T, TInput, TOutput>, IShardingConfiguration<T>)

public ZeRO3Optimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config)

Parameters