Table of Contents

Class SpikingLayer<T>

Namespace
AiDotNet.NeuralNetworks.Layers
Assembly
AiDotNet.dll

Represents a layer of spiking neurons that model the biological dynamics of neural activity.

public class SpikingLayer<T> : LayerBase<T>, ILayer<T>, IJitCompilable<T>, IDiagnosticsProvider, IWeightLoadable<T>, IDisposable

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
SpikingLayer<T>
Implements
Inherited Members

Remarks

A spiking layer implements various biologically-inspired neuron models that operate with discrete spike events rather than continuous activation values. The layer supports several neuron types including Leaky Integrate-and-Fire, Izhikevich, Adaptive Exponential, and Hodgkin-Huxley models. Spiking neurons are characterized by their membrane potential dynamics, threshold-crossing spike generation, and refractory periods after firing.

For Beginners: This layer mimics how real neurons in the brain work, using "spikes" instead of smooth values.

Think of each neuron as a tiny battery that:

  • Builds up electrical charge over time (membrane potential)
  • Fires a signal (spike) when the charge reaches a threshold
  • Needs a short recovery period after firing (refractory period)
  • Has different ways of charging and discharging (different neuron models)

Benefits include:

  • More biologically realistic modeling of neural activity
  • Potential for energy-efficient computation (spikes are sparse)
  • Ability to process time-dependent information naturally

For example, spiking neurons can directly model the timing patterns in speech or detect motion in video by responding to when things change rather than constantly processing every value.

Constructors

SpikingLayer(int, int, SpikingNeuronType, double, double)

Initializes a new instance of the SpikingLayer<T> class with the specified dimensions and neuron type.

public SpikingLayer(int inputSize, int outputSize, SpikingNeuronType neuronType = SpikingNeuronType.LeakyIntegrateAndFire, double tau = 10, double refractoryPeriod = 2)

Parameters

inputSize int

The number of input neurons.

outputSize int

The number of output neurons.

neuronType SpikingNeuronType

The type of spiking neuron model to use. Defaults to LeakyIntegrateAndFire.

tau double

Time constant for membrane potential decay. Defaults to 10.0.

refractoryPeriod double

Refractory period in time steps. Defaults to 2.0.

Remarks

This constructor creates a spiking layer with the specified input and output dimensions, neuron type, and parameters. It initializes the weights with small random values, and sets up the appropriate state variables for the selected neuron model.

For Beginners: This sets up a new layer of spiking neurons with your chosen settings.

When creating a spiking layer, you need to specify:

  • inputSize: How many input values the layer receives
  • outputSize: How many spiking neurons to create
  • neuronType: Which mathematical model to use (default: LeakyIntegrateAndFire)
  • tau: How quickly the membrane potential decays (higher = slower decay)
  • refractoryPeriod: How long neurons rest after firing

The constructor automatically initializes all the internal variables needed for the specified neuron type, with appropriate default values.

Properties

ParameterCount

Gets the total number of trainable parameters in the layer.

public override int ParameterCount { get; }

Property Value

int

The total number of weights and biases in the layer.

Remarks

This property returns the total number of trainable parameters in the layer, which is the sum of the number of weights and the number of biases.

For Beginners: This tells you how many adjustable numbers the layer has.

The parameter count is:

  • Number of weights (connections between input and output)
  • Plus the number of biases (one per output neuron)

This gives you an idea of the layer's complexity and memory requirements.

SupportsGpuExecution

Gets a value indicating whether this layer supports GPU execution.

protected override bool SupportsGpuExecution { get; }

Property Value

bool

SupportsJitCompilation

Gets a value indicating whether this layer supports JIT compilation.

public override bool SupportsJitCompilation { get; }

Property Value

bool

Always true. SpikingLayer uses surrogate gradients for JIT compilation.

Remarks

JIT compilation for spiking neurons uses a surrogate gradient approach where the non-differentiable spike threshold is approximated with a smooth function during backpropagation. The forward pass produces discrete spikes (0 or 1), but gradients are computed using a sigmoid-based surrogate.

SupportsTraining

Gets a value indicating whether this layer supports training through backpropagation.

public override bool SupportsTraining { get; }

Property Value

bool

Always returns true as spiking layers have trainable parameters.

Remarks

This property indicates that the spiking layer can be trained. The layer contains trainable parameters (weights and biases) that are updated during the training process.

For Beginners: This property tells you that the layer can learn from data.

A value of true means:

  • The layer contains numbers (parameters) that can be adjusted during training
  • It will improve its performance as it sees more examples
  • It participates in the learning process of the neural network

This is important because training spiking neural networks can be challenging due to the non-differentiable nature of spikes, but this layer implements special techniques to make it possible.

Methods

Backward(Tensor<T>)

Performs the backward pass of the spiking layer.

public override Tensor<T> Backward(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

The gradient of the loss with respect to the layer's output.

Returns

Tensor<T>

The gradient of the loss with respect to the layer's input.

Remarks

This method implements the backward pass of the spiking layer, which is used during training to propagate error gradients back through the network. Since spike events are non-differentiable, it uses a surrogate gradient approach to approximate the derivative of the spike function. This enables backpropagation through spiking neurons by providing a smooth approximation to the discontinuous threshold crossing.

For Beginners: This method calculates how to adjust the network during training.

Training spiking neurons is challenging because spikes are binary events (either a neuron fires or it doesn't), which normally can't be used with standard neural network training methods.

To solve this problem, this method:

  1. Uses a technique called "surrogate gradients" to approximate how changes in membrane potential affect spiking
  2. It computes a smooth function based on the membrane potential that approximates spike probability
  3. Using this approximation, it calculates:
    • How to adjust weights and biases to improve performance
    • How errors should flow back to previous layers

This clever approach allows spiking neural networks to be trained with backpropagation despite their fundamentally discrete nature.

Deserialize(BinaryReader)

Deserializes the layer's parameters and state from a binary stream.

public override void Deserialize(BinaryReader reader)

Parameters

reader BinaryReader

The binary reader to read from.

Remarks

This method reads the layer's parameters from a binary stream, including neuron type, time constants, weights, biases, and model-specific parameters. This allows a previously saved layer to be loaded from disk. It also initializes any model-specific variables needed for the selected neuron type.

For Beginners: This method loads the layer's configuration and parameters from a file.

The deserialization reads:

  • Basic parameters (neuron type, time constant, refractory period)
  • All weights and biases
  • Model-specific parameters for the particular neuron type

It also initializes any special variables needed for the specific neuron model. This lets you load a previously saved model and continue using or training it.

ExportComputationGraph(List<ComputationNode<T>>)

Exports the layer's computation graph for JIT compilation.

public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)

Parameters

inputNodes List<ComputationNode<T>>

List to populate with input computation nodes.

Returns

ComputationNode<T>

The output computation node representing the layer's operation.

Remarks

This method constructs a computation graph representation of the layer's forward pass that can be JIT compiled for faster inference. All layers MUST implement this method to support JIT compilation.

For Beginners: JIT (Just-In-Time) compilation converts the layer's operations into optimized native code for 5-10x faster inference.

To support JIT compilation, a layer must:

  1. Implement this method to export its computation graph
  2. Set SupportsJitCompilation to true
  3. Use ComputationNode and TensorOperations to build the graph

All layers are required to implement this method, even if they set SupportsJitCompilation = false.

Forward(Tensor<T>)

Performs the forward pass of the spiking layer.

public override Tensor<T> Forward(Tensor<T> input)

Parameters

input Tensor<T>

The input tensor to process.

Returns

Tensor<T>

The output tensor containing spike events.

Remarks

This method implements the forward pass of the spiking layer. It processes the input through the appropriate neuron model, updates the membrane potentials, and generates output spikes when the potentials cross their thresholds.

For Beginners: This method processes data through the spiking neurons.

During the forward pass:

  1. The input is saved for later use in training
  2. The input is converted to a vector for easier processing
  3. The input is processed through the specific neuron model (LIF, Izhikevich, etc.)
  4. This updates the membrane potentials and generates spikes when thresholds are crossed
  5. The resulting spikes (0 or 1 values) are returned as the output

This is how the layer transforms input signals into spikes, mimicking how real neurons convert inputs into action potentials.

ForwardGpu(params IGpuTensor<T>[])

Performs the GPU-accelerated forward pass for spiking neurons.

public override IGpuTensor<T> ForwardGpu(params IGpuTensor<T>[] inputs)

Parameters

inputs IGpuTensor<T>[]

The GPU tensor inputs. First element is the input activation.

Returns

IGpuTensor<T>

A GPU tensor containing the spike outputs.

Remarks

Spiking neurons maintain complex stateful dynamics (membrane potential, refractory periods) that require sequential updates. This method uses GPU for input/output transfer while processing neuron dynamics on CPU due to their inherently sequential nature.

GetParameters()

Gets all trainable parameters of the layer as a single vector.

public override Vector<T> GetParameters()

Returns

Vector<T>

A vector containing all trainable parameters.

Remarks

This method retrieves all trainable parameters of the layer (weights and biases) and combines them into a single vector. This is useful for optimization algorithms that operate on all parameters at once, or for saving and loading model weights.

For Beginners: This method collects all the learnable values from the layer into a single list.

The returned vector contains:

  1. All weight values (rows * columns), stored row by row
  2. All bias values (one per output neuron)

This method is useful for:

  • Saving the model to disk
  • Loading parameters from a previously trained model
  • Advanced optimization techniques that need access to all parameters

ResetState()

Resets the internal state of the spiking layer.

public override void ResetState()

Remarks

This method resets all internal state variables of the layer, including membrane potentials, refractory periods, and model-specific variables. It also clears cached inputs and outputs, and resets gradient accumulators. This is useful when starting a new sequence or when implementing stateful recurrent networks.

For Beginners: This method clears the layer's memory and state to start fresh.

When resetting the state:

  • All neuron membrane potentials are set to zero
  • Refractory period counters are reset
  • All spikes are cleared
  • Model-specific variables are reset to their initial values
  • Cached inputs and outputs are cleared
  • Gradient accumulators for training are reset

This is important for:

  • Processing a new sequence that's unrelated to the previous one
  • Preventing information from one sequence affecting another
  • Starting a new training episode
  • Resetting after a period of high activity

Serialize(BinaryWriter)

Serializes the layer's parameters and state to a binary stream.

public override void Serialize(BinaryWriter writer)

Parameters

writer BinaryWriter

The binary writer to write to.

Remarks

This method writes the layer's parameters, including neuron type, time constants, weights, biases, and model-specific parameters, to a binary stream. This allows the layer to be saved to disk for later use.

For Beginners: This method saves the layer's configuration and parameters to a file.

The serialization includes:

  • Basic parameters (neuron type, time constant, refractory period)
  • All weights and biases
  • Model-specific parameters like those for Izhikevich or AdEx models

This allows you to:

  • Save a trained model to disk
  • Load it later for inference or continued training
  • Transfer the model to another application

UpdateParameters(Vector<T>)

Updates the layer parameters with new values.

public override void UpdateParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

The new parameter values as a flat vector.

Remarks

This method updates the weights and biases of the layer from a flat vector of parameters. It assumes the vector contains all weights followed by all biases.

For Beginners: This method loads new values into the layer's weights and biases.

The parameter vector is expected to contain:

  1. All weight values (rows * columns), stored row by row
  2. All bias values (one per output neuron)

This method is useful for:

  • Loading a previously saved model
  • Setting up the layer with pre-determined weights
  • Testing with specific parameter values

UpdateParameters(T)

Updates the parameters of the layer using the calculated gradients and learning rate.

public override void UpdateParameters(T learningRate)

Parameters

learningRate T

The learning rate to use for the parameter updates.

Remarks

This method updates the weights and biases of the layer based on the gradients accumulated during the backward pass. It applies the learning rate to determine the size of the updates, subtracts the scaled gradients from the current parameters, and then resets the gradients for the next batch.

For Beginners: This method applies the calculated adjustments to improve the network.

After calculating how the weights and biases should change in the backward pass:

  1. The method applies these changes using the learning rate to control their size
  2. For each weight and bias:
    • Compute the update as learning rate × gradient
    • Subtract this update from the current value (moving in the opposite direction of the gradient)
    • Reset the gradient accumulator to zero for the next batch

The learning rate is important:

  • Smaller values mean smaller updates (more stable but slower learning)
  • Larger values mean larger updates (faster learning but potential instability)

This process is how the network gradually improves its performance over many training examples.

Edit this page