Table of Contents

Class LogVarianceLayer<T>

Namespace
AiDotNet.NeuralNetworks.Layers
Assembly
AiDotNet.dll

Represents a layer that computes the logarithm of variance along a specified axis in the input tensor.

public class LogVarianceLayer<T> : LayerBase<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
LogVarianceLayer<T>
Implements
Inherited Members

Remarks

The LogVarianceLayer calculates the statistical variance of values along a specified axis of the input tensor, and then computes the natural logarithm of that variance. This is often used in neural networks for calculating statistical measures, normalizing data, or as part of variational autoencoders (VAEs).

For Beginners: This layer measures how much the values in your data spread out from their average (variance), and then takes the logarithm of that spread.

Think of it like measuring how consistent or varied your data is:

  • Low values mean the data points are very similar to each other
  • High values mean the data points vary widely

For example, if you have a set of images:

  • Images that are very similar would produce low log-variance
  • Images that are very different would produce high log-variance

This is often used in AI models that need to understand the variation in the data, such as in models that generate new data similar to what they've been trained on.

Constructors

LogVarianceLayer(int[], int)

Initializes a new instance of the LogVarianceLayer<T> class.

public LogVarianceLayer(int[] inputShape, int axis)

Parameters

inputShape int[]

The shape of the input tensor.

axis int

The axis along which to calculate variance.

Remarks

This constructor creates a LogVarianceLayer that will calculate the variance along the specified axis of the input tensor. The output shape is determined by removing the specified axis from the input shape.

For Beginners: This creates a new log-variance layer with your desired settings.

When setting up this layer:

  • inputShape defines the expected size and dimensions of your data
  • axis specifies which dimension to calculate variance along

The layer will reduce the data along the specified axis, meaning the output will have one fewer dimension than the input.

Properties

Axis

Gets the axis along which the variance is calculated.

public int Axis { get; }

Property Value

int

Remarks

This property indicates the dimension of the input tensor along which the variance will be calculated. For example, if Axis is 1 and the input tensor has shape [batch, features], the variance will be calculated across the features dimension, resulting in one variance value per batch item.

For Beginners: This tells the layer which direction to look when calculating variance.

For example, with a 2D data array (like a table):

  • Axis 0 means calculate variance down each column
  • Axis 1 means calculate variance across each row

The value depends on how your data is organized and what kind of variance you want to measure.

SupportsGpuExecution

Gets a value indicating whether this layer supports GPU execution.

protected override bool SupportsGpuExecution { get; }

Property Value

bool

SupportsJitCompilation

Gets whether this layer supports JIT compilation.

public override bool SupportsJitCompilation { get; }

Property Value

bool

True if the layer can be JIT compiled, false otherwise.

Remarks

This property indicates whether the layer has implemented ExportComputationGraph() and can benefit from JIT compilation. All layers MUST implement this property.

For Beginners: JIT compilation can make inference 5-10x faster by converting the layer's operations into optimized native code.

Layers should return false if they:

  • Have not yet implemented a working ExportComputationGraph()
  • Use dynamic operations that change based on input data
  • Are too simple to benefit from JIT compilation

When false, the layer will use the standard Forward() method instead.

SupportsTraining

Gets a value indicating whether this layer supports training through backpropagation.

public override bool SupportsTraining { get; }

Property Value

bool

Remarks

This property returns false because the LogVarianceLayer does not have any trainable parameters, though it does support backward pass for gradient propagation through the network.

For Beginners: This tells you if the layer can learn from training data.

A value of false means:

  • This layer doesn't have any values that get updated during training
  • It performs a fixed mathematical calculation (log of variance)
  • However, during training, it still helps gradients flow backward through the network

Methods

Backward(Tensor<T>)

Performs the backward pass of the log-variance layer.

public override Tensor<T> Backward(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

The gradient of the loss with respect to the layer's output.

Returns

Tensor<T>

The gradient of the loss with respect to the layer's input.

Remarks

This method implements the backward pass of the log-variance layer, which is used during training to propagate error gradients backward through the network. It calculates how changes in the output affect the input, taking into account the derivatives of the logarithm and variance calculations.

For Beginners: This method is used during training to calculate how changes in the output would affect the input.

During the backward pass:

  • The layer receives information about how its output affected the overall error
  • It calculates how each input value contributed to that error
  • This information is passed backward to earlier layers

The mathematics here are complex but involve the chain rule from calculus:

  • For the log function: the derivative is 1/x
  • For variance: it involves how each value's difference from the mean contributed

This process is part of how neural networks learn from their mistakes.

BackwardGpu(IGpuTensor<T>)

GPU-accelerated backward pass for the log-variance layer.

public override IGpuTensor<T> BackwardGpu(IGpuTensor<T> outputGradient)

Parameters

outputGradient IGpuTensor<T>

The gradient of the loss with respect to the layer's output.

Returns

IGpuTensor<T>

GPU-resident gradient of the loss with respect to the layer's input.

Remarks

The gradient for log(variance + epsilon) is computed as: d(loss)/d(input) = outputGradient * (2 * (input - mean) / (axis_size * (variance + epsilon)))

ExportComputationGraph(List<ComputationNode<T>>)

Exports the layer's computation graph for JIT compilation.

public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)

Parameters

inputNodes List<ComputationNode<T>>

List to populate with input computation nodes.

Returns

ComputationNode<T>

The output computation node representing the layer's operation.

Remarks

This method constructs a computation graph representation of the layer's forward pass that can be JIT compiled for faster inference. All layers MUST implement this method to support JIT compilation.

For Beginners: JIT (Just-In-Time) compilation converts the layer's operations into optimized native code for 5-10x faster inference.

To support JIT compilation, a layer must:

  1. Implement this method to export its computation graph
  2. Set SupportsJitCompilation to true
  3. Use ComputationNode and TensorOperations to build the graph

All layers are required to implement this method, even if they set SupportsJitCompilation = false.

Forward(Tensor<T>)

Performs the forward pass of the log-variance layer.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

The input tensor.

Returns

Tensor<T>

A tensor containing the log-variance values.

Remarks

This method implements the forward pass of the log-variance calculation. It first computes the mean along the specified axis, then calculates the variance by summing squared differences from the mean, and finally takes the natural logarithm of the variance (with a small epsilon added for numerical stability).

For Beginners: This method processes your data through the layer, calculating the log-variance.

The calculation happens in these steps:

  1. Calculate the average (mean) of values along the specified axis
  2. For each value, find how far it is from the average
  3. Square these differences and add them up
  4. Divide by the number of values to get the variance
  5. Take the natural logarithm of the variance

A small value (epsilon) is added to prevent errors when taking the logarithm of zero or very small numbers.

ForwardGpu(params IGpuTensor<T>[])

Performs GPU-accelerated forward pass for log-variance reduction.

public override IGpuTensor<T> ForwardGpu(params IGpuTensor<T>[] inputs)

Parameters

inputs IGpuTensor<T>[]

Input GPU tensors (uses first input).

Returns

IGpuTensor<T>

GPU-resident output tensor with log-variance values.

GetParameters()

Gets all trainable parameters of the layer as a single vector.

public override Vector<T> GetParameters()

Returns

Vector<T>

An empty vector since this layer has no trainable parameters.

Remarks

This method returns an empty vector because the LogVarianceLayer has no trainable parameters. However, it must be implemented to satisfy the base class contract.

For Beginners: This method would normally return all the values that can be learned during training.

Since this layer has no learnable values:

  • It returns an empty list (vector with length 0)
  • This is expected for mathematical operation layers
  • Other layers, like those with weights, would return those weights

ResetState()

Resets the internal state of the layer.

public override void ResetState()

Remarks

This method clears any cached data from previous forward passes, essentially resetting the layer to its initial state. This is useful when starting to process a new batch of data or when implementing recurrent neural networks.

For Beginners: This method clears the layer's memory to start fresh.

When resetting the state:

  • Stored inputs and calculated values are cleared
  • The layer forgets any information from previous data
  • This is important when processing a new, unrelated batch of data

Think of it like wiping a calculator's memory before starting a new calculation.

UpdateParameters(T)

Updates the parameters of the layer based on the calculated gradients.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate to use for the parameter updates.

Remarks

This method is empty because the LogVarianceLayer has no trainable parameters to update. However, it must be implemented to satisfy the base class contract.

For Beginners: This method would normally update the layer's internal values during training.

However, since this layer doesn't have any trainable parameters:

  • There's nothing to update
  • The method exists but doesn't do anything
  • This is normal for layers that perform fixed mathematical operations
Edit this page