Class SpectralNormalizationLayer<T>
- Namespace
- AiDotNet.NeuralNetworks.Layers
- Assembly
- AiDotNet.dll
Represents a spectral normalization layer that normalizes the weights of a layer by their spectral norm.
public class SpectralNormalizationLayer<T> : LayerBase<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>, IDisposableType Parameters
TThe numeric type used for calculations, typically float or double.
- Inheritance
-
LayerBase<T>SpectralNormalizationLayer<T>
- Implements
-
ILayer<T>
- Inherited Members
Remarks
Spectral normalization is a weight normalization technique that constrains the Lipschitz constant of a neural network layer. It does this by dividing the weight matrix by its largest singular value (spectral norm). This technique is particularly effective for stabilizing GAN training.
For Beginners: Spectral normalization keeps layer weights from getting too large.
Key benefits:
- Stabilizes GAN training by preventing extreme weight values
- Ensures the discriminator doesn't become too powerful too quickly
- Helps prevent mode collapse in GANs
- Computationally efficient compared to other normalization methods
How it works:
- Computes the largest singular value of the weight matrix
- Divides all weights by this value
- Keeps weights normalized throughout training
Reference: Miyato et al., "Spectral Normalization for Generative Adversarial Networks" (2018)
Constructors
SpectralNormalizationLayer(ILayer<T>, int)
Initializes a new instance of the SpectralNormalizationLayer<T> class.
public SpectralNormalizationLayer(ILayer<T> innerLayer, int powerIterations = 1)Parameters
innerLayerILayer<T>The layer whose weights will be spectrally normalized.
powerIterationsintThe number of power iterations to perform. Default is 1.
Properties
SupportsGpuExecution
Gets a value indicating whether this layer supports GPU execution.
protected override bool SupportsGpuExecution { get; }Property Value
SupportsGpuTraining
Gets a value indicating whether this layer supports GPU-resident training. Delegates to the inner layer's capability.
public override bool SupportsGpuTraining { get; }Property Value
SupportsJitCompilation
Gets a value indicating whether this layer supports JIT compilation.
public override bool SupportsJitCompilation { get; }Property Value
Remarks
JIT compilation is supported if the inner layer supports it. At JIT export time, the spectral normalization is applied to create normalized weights, which are then used in the exported computation graph for inference.
SupportsTraining
Gets a value indicating whether this layer supports training.
public override bool SupportsTraining { get; }Property Value
Methods
Backward(Tensor<T>)
Performs backpropagation through the layer.
public override Tensor<T> Backward(Tensor<T> outputGradient)Parameters
outputGradientTensor<T>
Returns
- Tensor<T>
Remarks
Backpropagation uses the normalized weights (applied during Forward) to ensure gradients correspond to the actual weights used in the forward pass. After computing gradients, the original weights are restored.
BackwardGpu(IGpuTensor<T>)
GPU-resident backward pass for spectral normalization layer. Delegates to the inner layer's BackwardGpu method.
public override IGpuTensor<T> BackwardGpu(IGpuTensor<T> outputGradient)Parameters
outputGradientIGpuTensor<T>The gradient of the loss with respect to the layer's output (GPU tensor).
Returns
- IGpuTensor<T>
The gradient of the loss with respect to the layer's input (GPU tensor).
ExportComputationGraph(List<ComputationNode<T>>)
Exports the computation graph for JIT compilation.
public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)Parameters
inputNodesList<ComputationNode<T>>List to populate with input computation nodes.
Returns
- ComputationNode<T>
The output computation node representing the spectrally normalized layer.
Remarks
For JIT compilation, spectral normalization is applied at export time to produce normalized weights. These normalized weights are then used in the inner layer's computation graph. This approach is suitable for inference, where the weights are fixed after training.
Note: The exported computation graph uses a snapshot of the normalized weights at the time of export. If the underlying weights change, the graph must be re-exported to reflect those changes.
Forward(Tensor<T>)
Performs the forward pass through the layer with spectrally normalized weights.
public override Tensor<T> Forward(Tensor<T> input)Parameters
inputTensor<T>
Returns
- Tensor<T>
ForwardGpu(params IGpuTensor<T>[])
Performs the forward pass using GPU-resident tensors with GPU-accelerated spectral normalization.
public override IGpuTensor<T> ForwardGpu(params IGpuTensor<T>[] inputs)Parameters
inputsIGpuTensor<T>[]
Returns
- IGpuTensor<T>
A GPU-resident output tensor.
Remarks
This method performs spectral normalization using GPU-accelerated power iteration, keeping all computations on GPU for maximum performance.
GetParameterGradients()
Gets the parameter gradients from the inner layer.
public override Vector<T> GetParameterGradients()Returns
- Vector<T>
GetParameters()
Gets the parameters of the inner layer.
public override Vector<T> GetParameters()Returns
- Vector<T>
ResetState()
Resets the internal state of the layer.
public override void ResetState()SetParameters(Vector<T>)
Sets the parameters of the inner layer.
public override void SetParameters(Vector<T> parameters)Parameters
parametersVector<T>
UpdateParameters(T)
Updates the parameters of the inner layer.
public override void UpdateParameters(T learningRate)Parameters
learningRateT
UpdateParametersGpu(IGpuOptimizerConfig)
GPU-resident parameter update using the provided optimizer configuration. Delegates to the inner layer's UpdateParametersGpu method.
public override void UpdateParametersGpu(IGpuOptimizerConfig config)Parameters
configIGpuOptimizerConfigGPU optimizer configuration specifying the optimizer type and hyperparameters.