Table of Contents

Class SparseLinearLayer<T>

Namespace
AiDotNet.NeuralNetworks.Layers
Assembly
AiDotNet.dll

Represents a fully connected layer with sparse weight matrix for efficient computation.

public class SparseLinearLayer<T> : LayerBase<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations (float or double).

Inheritance
SparseLinearLayer<T>
Implements
Inherited Members

Remarks

A sparse linear layer is similar to a dense layer but uses sparse weight storage. This is efficient when most weights are zero (or can be pruned), reducing both memory usage and computation time.

For Beginners: This layer works like a regular dense layer, but uses sparse matrices to store weights more efficiently.

Benefits of sparse layers:

  • Much less memory for large layers with few active connections
  • Faster computation (only non-zero weights are used)
  • Natural for network pruning and compression

Use cases:

  • Graph neural networks (sparse adjacency)
  • Recommender systems (sparse user-item matrices)
  • Pruned neural networks
  • Very large embedding layers

Constructors

SparseLinearLayer(int, int, double, IActivationFunction<T>?)

Initializes a new instance of the SparseLinearLayer.

public SparseLinearLayer(int inputFeatures, int outputFeatures, double sparsity = 0.9, IActivationFunction<T>? activationFunction = null)

Parameters

inputFeatures int

Number of input features.

outputFeatures int

Number of output features.

sparsity double

Fraction of weights to be zero (0.0 to 1.0).

activationFunction IActivationFunction<T>

Optional activation function.

Properties

InputFeatures

Gets the number of input features.

public int InputFeatures { get; }

Property Value

int

OutputFeatures

Gets the number of output features.

public int OutputFeatures { get; }

Property Value

int

ParameterCount

Gets the total number of trainable parameters.

public override int ParameterCount { get; }

Property Value

int

Remarks

For sparse layers, this returns the number of non-zero weights plus biases.

SupportsJitCompilation

Gets whether this layer supports JIT compilation.

public override bool SupportsJitCompilation { get; }

Property Value

bool

Remarks

JIT compilation is supported by converting sparse weights to dense format at export time. This enables JIT compilation while preserving correct functionality, though the sparse memory benefits are not retained in the compiled graph. Returns true only if the activation function also supports JIT compilation.

SupportsTraining

Gets whether this layer supports training.

public override bool SupportsTraining { get; }

Property Value

bool

Methods

Backward(Tensor<T>)

Performs the backward pass through the layer.

public override Tensor<T> Backward(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

Gradient from the next layer.

Returns

Tensor<T>

Gradient to pass to the previous layer.

ExportComputationGraph(List<ComputationNode<T>>)

Exports the layer's forward pass as a JIT-compilable computation graph.

public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)

Parameters

inputNodes List<ComputationNode<T>>

List to populate with input computation nodes.

Returns

ComputationNode<T>

The output computation node representing the linear transformation.

Remarks

This method converts sparse weights to dense format at export time to enable JIT compilation. The resulting computation graph performs a standard dense matrix multiplication: output = input * W^T + bias

While this approach loses the memory benefits of sparse storage during inference, it ensures correct functionality and enables JIT optimization of the compiled graph.

Forward(Tensor<T>)

Performs the forward pass through the layer.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

Input tensor with shape [inputFeatures] or [batch, inputFeatures].

Returns

Tensor<T>

Output tensor with shape [outputFeatures] or [batch, outputFeatures].

GetParameters()

Gets all trainable parameters as a single vector.

public override Vector<T> GetParameters()

Returns

Vector<T>

A vector containing all non-zero weights and biases.

ResetState()

Resets the internal state of the layer.

public override void ResetState()

SetParameters(Vector<T>)

Sets all trainable parameters from a single vector.

public override void SetParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

A vector containing all parameters to set.

UpdateParameters(T)

Updates the layer's parameters using the calculated gradients. Maintains sparsity pattern during updates.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate to use for the update.

Edit this page