Table of Contents

Class Autoencoder<T>

Namespace
AiDotNet.NeuralNetworks
Assembly
AiDotNet.dll

Represents an autoencoder neural network that can compress data into a lower-dimensional representation and reconstruct it.

public class Autoencoder<T> : NeuralNetworkBase<T>, INeuralNetworkModel<T>, INeuralNetwork<T>, IFullModel<T, Tensor<T>, Tensor<T>>, IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Tensor<T>, Tensor<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>, IGradientComputable<T, Tensor<T>, Tensor<T>>, IJitCompilable<T>, IInterpretableModel<T>, IInputGradientComputable<T>, IDisposable, IAuxiliaryLossLayer<T>, IDiagnosticsProvider

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
Autoencoder<T>
Implements
IFullModel<T, Tensor<T>, Tensor<T>>
IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>
IParameterizable<T, Tensor<T>, Tensor<T>>
ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>
IGradientComputable<T, Tensor<T>, Tensor<T>>
Inherited Members
Extension Methods

Remarks

An autoencoder is a type of neural network designed to learn efficient data encodings in an unsupervised manner. It consists of an encoder that compresses the input data into a lower-dimensional representation (the latent space) and a decoder that reconstructs the original data from this representation. Autoencoders are trained to minimize the difference between the original input and the reconstructed output.

For Beginners: An autoencoder is like a sophisticated compression and decompression system.

Think of it like this:

  • The encoder part takes your original data (like an image) and compresses it into a smaller representation
  • The middle layer (latent space) holds this compressed version of your data
  • The decoder part takes this compressed version and tries to recreate the original data

For example, with images:

  • You might compress a 256x256 pixel image (65,536 values) into just 100 numbers
  • The network learns which features are most important to preserve
  • It then learns to reconstruct the image from only those 100 numbers

This is useful for:

  • Data compression
  • Noise reduction (by removing noise during reconstruction)
  • Feature learning (the compressed representation often contains meaningful features)
  • Anomaly detection (unusual data is reconstructed poorly)

Constructors

Autoencoder(NeuralNetworkArchitecture<T>, T, int, int, ILossFunction<T>?)

Initializes a new instance of the Autoencoder<T> class.

public Autoencoder(NeuralNetworkArchitecture<T> architecture, T learningRate, int epochs = 1, int batchSize = 32, ILossFunction<T>? lossFunction = null)

Parameters

architecture NeuralNetworkArchitecture<T>

The architecture specification for the autoencoder.

learningRate T

The learning rate for training the autoencoder.

epochs int

The number of training epochs to perform.

batchSize int

The batch size to use during training.

lossFunction ILossFunction<T>

The loss function to use for training. If null, Mean Squared Error will be used.

Remarks

This constructor creates an autoencoder with the specified architecture and training parameters. It initializes the layers according to the architecture specification or uses default layers if none are provided. It also sets the EncodedSize property based on the size of the middle layer.

For Beginners: This constructor creates a new autoencoder with the specified settings.

The architecture parameter determines the structure of your network, while:

  • learningRate controls how quickly the model learns (typically 0.001 to 0.1)
  • epochs specifies how many times to process the entire dataset (often 10-100)
  • batchSize determines how many examples to process at once (typically 32-256)

These parameters balance learning speed against stability and accuracy.

Properties

AuxiliaryLossWeight

Gets or sets the weight for the sparsity penalty. Default is 0.001. Typical range: 0.0001 to 0.01.

public T AuxiliaryLossWeight { get; set; }

Property Value

T

EncodedSize

Gets the size of the encoded representation (latent space).

public int EncodedSize { get; }

Property Value

int

Remarks

This property indicates the dimensionality of the encoded representation, which is the size of the output from the middle (bottleneck) layer of the autoencoder. This is typically smaller than the input size, forcing the autoencoder to learn a compressed representation of the data.

For Beginners: This is the size of the compressed representation.

For example:

  • If your input data has 1000 dimensions (like a 1000-pixel image)
  • And EncodedSize is 50
  • Then your data is being compressed to 5% of its original size

The smaller this value:

  • The more compression is happening
  • The more the network has to be selective about what information to keep
  • The more efficient but potentially less accurate the reconstruction becomes

UseAuxiliaryLoss

Gets or sets whether to use auxiliary loss (sparsity penalty) during training. Default is false. Enable for sparse autoencoders.

public bool UseAuxiliaryLoss { get; set; }

Property Value

bool

Methods

CalculateLoss(Tensor<T>, Tensor<T>)

Calculates the loss between the predicted output and the expected output.

protected T CalculateLoss(Tensor<T> predicted, Tensor<T> expected)

Parameters

predicted Tensor<T>

The predicted output tensor.

expected Tensor<T>

The expected output tensor.

Returns

T

The calculated loss value.

Remarks

This method calculates the loss between the predicted output and the expected output using the loss function specified during initialization. For autoencoders, this typically measures how well the network reconstructs the input data.

For Beginners: This method measures how different the reconstructed output is from the expected output.

It uses the loss function that was specified when creating the autoencoder to calculate:

  • How far off the reconstruction is from the original
  • A single number representing the total error

Lower values mean better reconstruction (less difference between original and reconstructed data). This value guides the training process to improve the network's performance.

CalculateOutputGradient(Tensor<T>, Tensor<T>)

Calculates the gradient of the loss function with respect to the network output.

protected Tensor<T> CalculateOutputGradient(Tensor<T> predicted, Tensor<T> expected)

Parameters

predicted Tensor<T>

The predicted output tensor.

expected Tensor<T>

The expected output tensor.

Returns

Tensor<T>

The gradient tensor.

Remarks

This method calculates the gradient of the loss function with respect to the network output. This gradient is then used as the starting point for backpropagation to update the network weights.

For Beginners: This method calculates how to adjust the network to reduce the reconstruction error.

It works by:

  • Comparing the predicted output with the expected output
  • Calculating how the output needs to change to reduce the error
  • Creating a "gradient" that shows the direction and amount of recommended change

This gradient is then used to update the network's parameters during training, helping the autoencoder gradually improve its reconstruction ability.

ComputeAuxiliaryLoss()

Computes the auxiliary loss for sparse autoencoders, which penalizes non-sparse activations.

public T ComputeAuxiliaryLoss()

Returns

T

The sparsity loss value.

Remarks

The sparsity loss uses KL divergence between the target sparsity and actual average activation. Formula: KL(ρ || ρ̂) = ρ * log(ρ/ρ̂) + (1-ρ) * log((1-ρ)/(1-ρ̂)) where ρ is target sparsity and ρ̂ is actual average activation.

For Beginners: This calculates a penalty for having too many neurons active at once.

The sparsity loss:

  • Measures how far the actual neuron activations are from the target sparsity
  • Encourages only a few neurons to be active for each input
  • Forces the autoencoder to learn more selective, meaningful features
  • Results in better feature learning and interpretability

Without sparsity:

  • All neurons might activate for every input
  • Features become less distinct and harder to interpret
  • The autoencoder might just learn to memorize rather than extract key features

CreateNewInstance()

Creates a new instance of the autoencoder model.

protected override IFullModel<T, Tensor<T>, Tensor<T>> CreateNewInstance()

Returns

IFullModel<T, Tensor<T>, Tensor<T>>

A new instance of the autoencoder model with the same configuration.

Remarks

This method creates a new instance of the autoencoder model with the same configuration as the current instance. It is used internally during serialization/deserialization processes to create a fresh instance that can be populated with the serialized data. The new instance will have the same architecture, learning rate, epochs, batch size, and loss function as the original.

For Beginners: This method creates a copy of the model structure without copying the learned data.

Think of it like creating a blueprint of the autoencoder:

  • It copies the same overall design (how many layers, how they're arranged)
  • It preserves settings like learning rate and batch size
  • It keeps the same encoded size (compression level)
  • But it doesn't copy any of the learned knowledge yet

This is primarily used when saving or loading models, creating a framework that the saved parameters can be loaded into later.

Decode(Tensor<T>)

Decodes the latent space representation back to the original space.

public Tensor<T> Decode(Tensor<T> encodedInput)

Parameters

encodedInput Tensor<T>

The encoded tensor (latent space representation) to decode.

Returns

Tensor<T>

The reconstructed tensor in the original space.

Remarks

This method performs the decoding part of the autoencoder, passing the latent space representation through the decoder layers to reconstruct the original data. It starts from the middle layer of the autoencoder.

For Beginners: This method reconstructs your original data from the compressed representation.

When you call Decode:

  • Your compressed data is processed by the second half of the network (the decoder part)
  • It passes through progressively larger layers
  • It expands the compressed representation back to the original size
  • You get back a reconstruction of the original data

This is useful when you:

  • Want to see what information was preserved during compression
  • Need to generate new data from the latent space
  • Want to remove noise or fill in missing data

For example, you might create a new face image by decoding a point in the latent space.

DeserializeNetworkSpecificData(BinaryReader)

Deserializes network-specific data for the Autoencoder.

protected override void DeserializeNetworkSpecificData(BinaryReader reader)

Parameters

reader BinaryReader

The BinaryReader to read the data from.

Remarks

This method reads the specific configuration and state of the Autoencoder from a binary stream. It reconstructs the network-specific parameters to match the state of the network when it was serialized.

For Beginners: This method loads the unique settings of your Autoencoder.

It reads:

  • The encoded size (size of the compressed representation)
  • The learning rate used for training
  • The number of epochs and batch size used in training
  • The configuration of each layer

Loading these details allows you to recreate the exact same network structure that was previously saved. It's like following a detailed recipe to recreate a dish exactly as it was made before.

Encode(Tensor<T>)

Encodes the input data into the latent space representation.

public Tensor<T> Encode(Tensor<T> input)

Parameters

input Tensor<T>

The input tensor to encode.

Returns

Tensor<T>

The encoded tensor (latent space representation).

Remarks

This method performs the encoding part of the autoencoder, passing the input data through the encoder layers to produce the latent space representation. It stops at the middle layer of the autoencoder.

For Beginners: This method compresses your data into the smaller representation.

When you call Encode:

  • Your data is processed by the first half of the network (the encoder part)
  • It passes through progressively smaller layers
  • It stops at the middle layer (the bottleneck)
  • You get back the compressed representation

This is useful when you:

  • Want the compressed representation for other tasks
  • Need to reduce the dimensionality of your data
  • Want to extract the key features learned by the autoencoder

For example, you might encode images into a small vector to cluster similar images together.

GenerateSamples(int, double, double)

Generates new data samples by sampling points in the latent space and decoding them.

public Tensor<T> GenerateSamples(int count, double mean = 0, double stdDev = 1)

Parameters

count int

The number of samples to generate.

mean double

The mean value for random sampling (typically 0).

stdDev double

The standard deviation for random sampling (typically 1).

Returns

Tensor<T>

A tensor containing the generated samples.

Remarks

This method generates new data samples by randomly sampling points in the latent space and decoding them. It's useful for creative applications and exploring the learned data manifold.

For Beginners: This method creates new data samples similar to what the autoencoder has seen.

It works by:

  • Creating random points in the compressed representation space
  • Passing these points through the decoder part of the network
  • Producing new, synthetic data samples that resemble the training data

This can be used for:

  • Generating new content (like images or music)
  • Data augmentation (creating additional training examples)
  • Exploring what the autoencoder has learned

GetAuxiliaryLossDiagnostics()

Gets diagnostic information about the sparsity loss.

public Dictionary<string, string> GetAuxiliaryLossDiagnostics()

Returns

Dictionary<string, string>

A dictionary containing diagnostic information about sparsity.

Remarks

This method provides insights into the autoencoder's sparsity behavior, including: - Current sparsity loss value - Average activation level - Target sparsity parameter - Whether auxiliary loss is enabled

For Beginners: This gives you information to track sparsity during training.

The diagnostics include:

  • Sparsity Loss: How far from the target sparsity (lower is better)
  • Average Activation: Current average neuron activity level
  • Target Sparsity: The desired average activity level
  • Sparsity Weight: How much the sparsity penalty influences training

These values help you:

  • Verify that sparsity is being enforced
  • Tune the sparsity parameter and weight
  • Detect if neurons are dying (too much sparsity) or too active (too little)
  • Monitor training health and feature learning quality

GetDiagnostics()

Gets diagnostic information about this component's state and behavior. Overrides GetDiagnostics() to include auxiliary loss diagnostics.

public Dictionary<string, string> GetDiagnostics()

Returns

Dictionary<string, string>

A dictionary containing diagnostic metrics including both base layer diagnostics and auxiliary loss diagnostics from GetAuxiliaryLossDiagnostics().

GetModelMetadata()

Gets metadata about the autoencoder model.

public override ModelMetadata<T> GetModelMetadata()

Returns

ModelMetadata<T>

A ModelMetaData object containing information about the model.

Remarks

This method returns metadata about the autoencoder, including the model type, input/output dimensions, encoded size, and layer configuration. This information is useful for model management, serialization, and transfer learning.

InitializeLayers()

Initializes the layers of the autoencoder.

protected override void InitializeLayers()

Remarks

This method initializes the layers of the autoencoder either by using the layers provided by the user in the architecture specification or by creating default autoencoder layers if none are provided. It also sets the EncodedSize property based on the size of the middle layer.

For Beginners: This method sets up the building blocks of the autoencoder.

It does one of two things:

  1. If you provided specific layers in the architecture, it uses those
  2. If you didn't provide layers, it creates a default set of layers appropriate for an autoencoder

The default layers typically create a "bottleneck" shape:

  • Starting with larger layers (the input)
  • Getting progressively smaller (the encoder)
  • Reaching a small middle layer (the latent space)
  • Getting progressively larger again (the decoder)
  • Ending with a layer the same size as the input (the output)

It also sets the EncodedSize property to the size of the middle layer.

Predict(Tensor<T>)

Makes a prediction using the autoencoder by encoding and then decoding the input.

public override Tensor<T> Predict(Tensor<T> input)

Parameters

input Tensor<T>

The input tensor to process.

Returns

Tensor<T>

The reconstructed output tensor.

Remarks

This method performs a full forward pass through the autoencoder, encoding the input into the latent space and then decoding it back to the original space. The goal of training is to make this reconstructed output as close as possible to the original input.

For Beginners: This method runs your data through the entire autoencoder (compress then reconstruct).

It's essentially a shortcut that:

  1. Calls Encode() to compress your data
  2. Then calls Decode() to reconstruct it

When the autoencoder is well-trained:

  • The output should closely resemble the input
  • But with some differences based on what the network learned was important

For example, if you feed in a noisy image, you might get back a cleaner version as the autoencoder learns to focus on the important features and ignore the noise.

SerializeNetworkSpecificData(BinaryWriter)

Serializes network-specific data for the Autoencoder.

protected override void SerializeNetworkSpecificData(BinaryWriter writer)

Parameters

writer BinaryWriter

The BinaryWriter to write the data to.

Remarks

This method writes the specific configuration and state of the Autoencoder to a binary stream. It includes network-specific parameters that are essential for later reconstruction of the network.

For Beginners: This method saves the unique settings of your Autoencoder.

It writes:

  • The encoded size (size of the compressed representation)
  • The learning rate used for training
  • The number of epochs and batch size used in training
  • The configuration of each layer

Saving these details allows you to recreate the exact same network structure later. It's like writing down a detailed recipe so you can make the same dish again in the future.

SetSparsityParameter(T)

Sets the target sparsity parameter for sparse autoencoder training.

public void SetSparsityParameter(T sparsity)

Parameters

sparsity T

Target average activation level (typically 0.01 to 0.1).

Remarks

The sparsity parameter controls how "active" the encoder's hidden units should be on average. Lower values enforce stronger sparsity (fewer active neurons), while higher values allow more neurons to be active.

For Beginners: This controls how selective the autoencoder is about which features to use.

Sparsity means:

  • Lower values (e.g., 0.01): Only 1% of neurons active on average - very selective, learns distinct features
  • Medium values (e.g., 0.05): 5% active - good balance (default)
  • Higher values (e.g., 0.1): 10% active - less sparse, more distributed representations

Sparse representations often learn more interpretable and meaningful features.

Train(Tensor<T>, Tensor<T>)

Trains the autoencoder on the provided data.

public override void Train(Tensor<T> input, Tensor<T> expectedOutput)

Parameters

input Tensor<T>

The input data to train on.

expectedOutput Tensor<T>

The expected output, typically the same as the input for standard autoencoders.

Remarks

This method trains the autoencoder to reproduce the input as its output. For standard autoencoders, the expectedOutput is typically the same as the input. For denoising autoencoders, the input might be a noisy version while expectedOutput is the clean version.

For Beginners: This method teaches the autoencoder to compress and reconstruct your data effectively.

During training:

  • The autoencoder processes your input data through all its layers
  • It compares the reconstructed output with the expected output
  • It calculates how to adjust its internal parameters to make the reconstruction better
  • It updates all parameters to gradually improve performance

For a standard autoencoder, the expected output is the same as the input (it learns to recreate what it sees). For specialized autoencoders like denoising autoencoders, the input could be noisy data while the expected output is clean data.

UpdateParameters(Vector<T>)

Updates the parameters of the autoencoder.

public override void UpdateParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

The parameters to update the network with.

Remarks

This method updates the parameters of each layer in the autoencoder with the provided parameter values. It distributes the parameters to each layer based on the number of parameters in each layer.

For Beginners: This method adjusts the network's internal values to improve its performance.

During training:

  • The learning algorithm calculates how the parameters should change
  • This method applies those changes to the actual parameters
  • Each layer gets its own portion of the parameter updates

Think of it like fine-tuning all the components of the autoencoder based on feedback:

  • The encoder learns better ways to compress the data
  • The decoder learns better ways to reconstruct the data
  • Together they improve at preserving the most important information

ValidateCustomLayers(List<ILayer<T>>)

Validates that the custom layers conform to autoencoder requirements.

protected override void ValidateCustomLayers(List<ILayer<T>> layers)

Parameters

layers List<ILayer<T>>

The list of layers to validate.

Remarks

This method validates that the provided layers conform to the requirements of an autoencoder: 1. There must be at least 3 layers (input, encoded, and output). 2. The input and output layers must have the same size. 3. The architecture must be symmetric. 4. The activation functions must be symmetric.

For Beginners: This method checks if the layers you provided will work for an autoencoder.

It makes sure:

  • You have at least 3 layers (input ? encoded ? output)
  • The input and output layers are the same size (since an autoencoder reconstructs its input)
  • The network is symmetric (decoder mirrors encoder)
  • The activation functions are symmetric (same functions used in corresponding encoder/decoder layers)

If any of these requirements aren't met, it shows an error explaining what's wrong. This helps ensure your autoencoder is structured correctly before you start training it.

Exceptions

ArgumentException

Thrown when the layers do not conform to autoencoder requirements.

Edit this page