Class InstanceNormalizationLayer<T>

Namespace

AiDotNet.NeuralNetworks.Layers

Assembly

AiDotNet.dll

Represents an Instance Normalization layer that normalizes each channel independently across spatial dimensions.

public class InstanceNormalizationLayer<T> : LayerBase<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance

object

LayerBase<T>

InstanceNormalizationLayer<T>

Implements

ILayer<T>

IJitCompilable<T>

IDiagnosticsProvider

IWeightLoadable<T>

IDisposable

Inherited Members

LayerBase<T>.Engine

LayerBase<T>.ScalarActivation

LayerBase<T>.VectorActivation

LayerBase<T>.UsingVectorActivation

LayerBase<T>.NumOps

LayerBase<T>.Random

LayerBase<T>.Parameters

LayerBase<T>.ParameterGradients

LayerBase<T>.InputShape

LayerBase<T>.InputShapes

LayerBase<T>.UpdateInputShape(int[])

LayerBase<T>.OutputShape

LayerBase<T>.IsTrainingMode

LayerBase<T>.InitializationStrategy

LayerBase<T>.IsInitialized

LayerBase<T>.InitializationLock

LayerBase<T>.EnsureInitialized()

LayerBase<T>.UseAutodiff

LayerBase<T>.SetTrainingMode(bool)

LayerBase<T>.GetParameterGradients()

LayerBase<T>.ClearGradients()

LayerBase<T>.GetInputShape()

LayerBase<T>.GetInputShapes()

LayerBase<T>.GetOutputShape()

LayerBase<T>.GetWeights()

LayerBase<T>.GetBiases()

LayerBase<T>.MapActivationToFused()

LayerBase<T>.SupportsGpuTraining

LayerBase<T>.CanExecuteOnGpu

LayerBase<T>.CanTrainOnGpu

LayerBase<T>.UpdateParametersGpu(IGpuOptimizerConfig)

LayerBase<T>.UploadWeightsToGpu()

LayerBase<T>.DownloadWeightsFromGpu()

LayerBase<T>.ZeroGradientsGpu()

LayerBase<T>.GetActivationTypes()

LayerBase<T>.Forward(params Tensor<T>[])

LayerBase<T>.ApplyActivation(Tensor<T>)

LayerBase<T>.ApplyActivation(Vector<T>)

LayerBase<T>.ActivateTensor(IActivationFunction<T>, Tensor<T>)

LayerBase<T>.ActivateTensor(IVectorActivationFunction<T>, Tensor<T>)

LayerBase<T>.CalculateInputShape(int, int, int)

LayerBase<T>.CalculateOutputShape(int, int, int)

LayerBase<T>.Clone()

LayerBase<T>.DerivativeTensor(IActivationFunction<T>, Tensor<T>)

LayerBase<T>.ApplyActivationDerivative(T, T)

LayerBase<T>.ApplyActivationDerivative(Tensor<T>, Tensor<T>)

LayerBase<T>.ComputeActivationJacobian(Vector<T>)

LayerBase<T>.ApplyActivationDerivative(Vector<T>, Vector<T>)

LayerBase<T>.UpdateParameters(Vector<T>)

LayerBase<T>.Serialize(BinaryWriter)

LayerBase<T>.Deserialize(BinaryReader)

LayerBase<T>.GetDiagnostics()

LayerBase<T>.ApplyActivationToGraph(ComputationNode<T>)

LayerBase<T>.CanActivationBeJitted()

LayerBase<T>.RegisterTrainableParameter(Tensor<T>, PersistentTensorRole)

LayerBase<T>.InvalidateTrainableParameter(Tensor<T>)

LayerBase<T>.HasGpuActivation()

LayerBase<T>.ApplyActivationForwardGpu(IDirectGpuBackend, IGpuBuffer, IGpuBuffer, int)

LayerBase<T>.ApplyActivationBackwardGpu(IDirectGpuBackend, IGpuBuffer, IGpuBuffer, IGpuBuffer, IGpuBuffer, int)

LayerBase<T>.GetFusedActivationType()

LayerBase<T>.ApplyGpuActivation(IDirectGpuBackend, IGpuBuffer, IGpuBuffer, int, FusedActivationType)

LayerBase<T>.ApplyGpuActivationBackward(IDirectGpuBackend, IGpuBuffer, IGpuBuffer, IGpuBuffer, IGpuBuffer, int, FusedActivationType, float)

LayerBase<T>.Dispose()

LayerBase<T>.Dispose(bool)

LayerBase<T>.WeightParameterName

LayerBase<T>.BiasParameterName

LayerBase<T>.SetWeights(Tensor<T>)

LayerBase<T>.SetBiases(Tensor<T>)

LayerBase<T>.GetParameterNames()

LayerBase<T>.TryGetParameter(string, out Tensor<T>)

LayerBase<T>.SetParameter(string, Tensor<T>)

LayerBase<T>.GetParameterShape(string)

LayerBase<T>.NamedParameterCount

LayerBase<T>.ValidateWeights(IEnumerable<string>, Func<string, string>)

LayerBase<T>.LoadWeights(Dictionary<string, Tensor<T>>, Func<string, string>, bool)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Instance Normalization normalizes each channel independently for each sample in the batch. Unlike Batch Normalization which computes statistics across the batch dimension, Instance Normalization computes statistics independently for each sample and each channel. This is essentially Group Normalization with numGroups = numChannels.

For Beginners: This layer helps stabilize training, especially for style transfer and image generation.

Think of Instance Normalization like adjusting the contrast of each color channel independently:

• Each channel (e.g., red, green, blue) is normalized on its own
• Each image in the batch is treated independently
• This removes instance-specific contrast information

Key advantages:

• Works well for style transfer and image generation
• Independent of batch size (works with batch size of 1)
• Removes instance-specific style information, making it ideal for style transfer

Common usage:

• Style transfer networks (separates content from style)
• GANs (Generative Adversarial Networks)
• Image-to-image translation

Constructors

InstanceNormalizationLayer(int, double, bool)

Initializes a new instance of the InstanceNormalizationLayer.

public InstanceNormalizationLayer(int numChannels, double epsilon = 1E-05, bool affine = true)

Parameters

numChannels int

Number of channels (features) to normalize.

epsilon double

Small constant for numerical stability. Defaults to 1e-5.

affine bool

Whether to include learnable affine parameters (gamma and beta). Defaults to true.

Exceptions

ArgumentOutOfRangeException

Thrown when numChannels is not positive.

Properties

Affine

Gets whether affine transformation (learnable gamma and beta) is enabled.

public bool Affine { get; }

Property Value

bool

NumChannels

Gets the number of channels this layer normalizes.

public int NumChannels { get; }

Property Value

int

ParameterCount

Gets the total number of trainable parameters.

public override int ParameterCount { get; }

Property Value

int

SupportsGpuExecution

Gets a value indicating whether this layer supports GPU execution.

protected override bool SupportsGpuExecution { get; }

Property Value

bool

SupportsJitCompilation

Gets a value indicating whether this layer supports JIT compilation.

public override bool SupportsJitCompilation { get; }

Property Value

bool

Remarks

Instance normalization supports JIT compilation by leveraging GroupNorm with numGroups = numChannels.

SupportsTraining

Gets a value indicating whether this layer supports training.

public override bool SupportsTraining { get; }

Property Value

bool

Methods

Backward(Tensor<T>)

Performs the backward pass of instance normalization.

public override Tensor<T> Backward(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

Gradient from the next layer.

Returns

Tensor<T>

Gradient with respect to the input.

BackwardGpu(IGpuTensor<T>)

Performs the backward pass using GPU-resident tensors.

public override IGpuTensor<T> BackwardGpu(IGpuTensor<T> outputGradient)

Parameters

outputGradient IGpuTensor<T>

GPU-resident gradient tensor.

Returns

IGpuTensor<T>

GPU-resident input gradient tensor.

ExportComputationGraph(List<ComputationNode<T>>)

Exports the computation graph for JIT compilation.

public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)

Parameters

inputNodes List<ComputationNode<T>>

List to populate with input computation nodes.

Returns

ComputationNode<T>

The output computation node representing the InstanceNormalization operation.

Remarks

This method builds a computation graph representing the Instance Normalization layer. Instance normalization is implemented as GroupNorm with numGroups = numChannels, meaning each channel is normalized independently across spatial dimensions.

Forward(Tensor<T>)

Performs the forward pass of instance normalization.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

Input tensor with shape [batch, channels, ...spatial].

Returns

Tensor<T>

Normalized output tensor with the same shape as input.

Remarks

Supports any-rank tensors:

• 2D [batch, channels] - normalizes each channel
• 3D [batch, channels, length] - normalizes over length dimension
• 4D [batch, channels, height, width] - normalizes over spatial dimensions
• 5D [batch, channels, D, H, W] - normalizes over all spatial dimensions
• ND - generalizes to any number of dimensions

ForwardGpu(params IGpuTensor<T>[])

Performs the forward pass of instance normalization on GPU tensors.

public override IGpuTensor<T> ForwardGpu(params IGpuTensor<T>[] inputs)

Parameters

inputs IGpuTensor<T>[]

GPU tensor inputs.

Returns

IGpuTensor<T>

GPU tensor output after normalization.

Remarks

This method uses the native GPU InstanceNorm operation for efficient normalization where each channel is normalized independently.

GetBeta()

Gets the beta (shift) parameters.

public Vector<T> GetBeta()

Returns

Vector<T>

GetBetaTensor()

Gets the beta (shift) parameters as a tensor.

public Tensor<T> GetBetaTensor()

Returns

Tensor<T>

GetEpsilon()

Gets the epsilon value used for numerical stability.

public T GetEpsilon()

Returns

T

GetGamma()

Gets the gamma (scale) parameters.

public Vector<T> GetGamma()

Returns

Vector<T>

GetGammaTensor()

Gets the gamma (scale) parameters as a tensor.

public Tensor<T> GetGammaTensor()

Returns

Tensor<T>

GetParameters()

Gets all trainable parameters as a single vector.

public override Vector<T> GetParameters()

Returns

Vector<T>

A vector containing gamma and beta parameters (if affine) or empty vector.

ResetState()

Resets the internal state of the layer.

public override void ResetState()

SetParameters(Vector<T>)

Sets all trainable parameters from a single vector.

public override void SetParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

A vector containing all parameters to set.

UpdateParameters(T)

Updates the layer's parameters using the calculated gradients.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate for parameter updates.