Table of Contents

Class GaussianSplatting<T>

Namespace
AiDotNet.NeuralRadianceFields.Models
Assembly
AiDotNet.dll
public class GaussianSplatting<T> : NeuralNetworkBase<T>, INeuralNetworkModel<T>, IInterpretableModel<T>, IInputGradientComputable<T>, IDisposable, IRadianceField<T>, INeuralNetwork<T>, IFullModel<T, Tensor<T>, Tensor<T>>, IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Tensor<T>, Tensor<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>, IGradientComputable<T, Tensor<T>, Tensor<T>>, IJitCompilable<T>

Type Parameters

T
Inheritance
GaussianSplatting<T>
Implements
IFullModel<T, Tensor<T>, Tensor<T>>
IModel<Tensor<T>, Tensor<T>, ModelMetadata<T>>
IParameterizable<T, Tensor<T>, Tensor<T>>
ICloneable<IFullModel<T, Tensor<T>, Tensor<T>>>
IGradientComputable<T, Tensor<T>, Tensor<T>>
Inherited Members
Extension Methods

Constructors

GaussianSplatting() 

public GaussianSplatting()

GaussianSplatting(GaussianSplattingOptions, Matrix<T>?, Matrix<T>?, ILossFunction<T>?) 

public GaussianSplatting(GaussianSplattingOptions options, Matrix<T>? initialPointCloud = null, Matrix<T>? initialColors = null, ILossFunction<T>? lossFunction = null)

Parameters

options GaussianSplattingOptions
initialPointCloud Matrix<T>
initialColors Matrix<T>
lossFunction ILossFunction<T>

GaussianSplatting(Matrix<T>?, Matrix<T>?, bool, int, ILossFunction<T>?)

Initializes a new instance of the GaussianSplatting class.

public GaussianSplatting(Matrix<T>? initialPointCloud = null, Matrix<T>? initialColors = null, bool useSphericalHarmonics = true, int shDegree = 3, ILossFunction<T>? lossFunction = null)

Parameters

initialPointCloud Matrix<T>

Initial point cloud to place Gaussians.

initialColors Matrix<T>

Optional initial colors for each point.

useSphericalHarmonics bool

Whether to use spherical harmonics for view-dependent appearance.

shDegree int

Degree of spherical harmonics (0-3, higher = more view dependence).

lossFunction ILossFunction<T>

Optional loss function for training.

Remarks

For Beginners: Creates a Gaussian Splatting model from an initial point cloud.

Parameters explained:

  • initialPointCloud: Starting 3D points (typically from SfM like COLMAP)

    • Format: Matrix [N, 3] where N is number of points
    • Each row: (x, y, z) position
    • Typical: 10K-1M points depending on scene

  • initialColors: Starting colors for each point

    • Format: Matrix [N, 3] with RGB values
    • Optional: If null, will be initialized randomly
    • Usually from SfM output or estimated

  • useSphericalHarmonics: Enable view-dependent appearance

    • False: Constant color from all viewing angles (faster, simpler)
    • True: Color changes with viewing direction (realistic, e.g., specular highlights)
    • Recommended: True for realistic scenes

  • shDegree: How much view-dependence to model

    • 0: No view dependence (constant color) - simplest
    • 1: Linear variation - basic view dependence
    • 2: Quadratic variation - moderate view dependence
    • 3: Cubic variation - strong view dependence (realistic)
    • Higher degree = more parameters per Gaussian
      • Degree 0: 3 parameters (RGB)
      • Degree 1: 3 + 9 = 12 parameters
      • Degree 2: 3 + 9 + 15 = 27 parameters
      • Degree 3: 3 + 9 + 15 + 21 = 48 parameters

Example initialization: // Load point cloud from COLMAP var pointCloud = LoadCOLMAPPointCloud("scene.ply"); var colors = LoadCOLMAPColors("scene.ply");

// Create Gaussian Splatting model var gs = new GaussianSplatting( initialPointCloud: pointCloud, initialColors: colors, useSphericalHarmonics: true, shDegree: 3 // High quality view-dependent effects );

Typical workflow:

  1. Run COLMAP on images → Get point cloud
  2. Initialize GaussianSplatting with point cloud
  3. Train on images for 10-30 minutes
  4. Render novel views at 100+ FPS

Properties

ColorLearningRate 

public double ColorLearningRate { get; set; }

Property Value

double

DefaultPointSpacing 

public double DefaultPointSpacing { get; set; }

Property Value

double

DensificationInterval 

public int DensificationInterval { get; set; }

Property Value

int

EnableDensification 

public bool EnableDensification { get; set; }

Property Value

bool

EnableSpatialIndex 

public bool EnableSpatialIndex { get; set; }

Property Value

bool

GaussianCount

Gets the number of Gaussians currently in the scene.

public int GaussianCount { get; }

Property Value

int

InitialNeighborSearchScale 

public double InitialNeighborSearchScale { get; set; }

Property Value

double

InitialScaleMultiplier 

public double InitialScaleMultiplier { get; set; }

Property Value

double

MaxGaussians 

public int MaxGaussians { get; set; }

Property Value

int

MinScale 

public double MinScale { get; set; }

Property Value

double

OpacityLearningRate 

public double OpacityLearningRate { get; set; }

Property Value

double

PositionLearningRate 

public double PositionLearningRate { get; set; }

Property Value

double

PruneOpacityThreshold 

public double PruneOpacityThreshold { get; set; }

Property Value

double

RotationLearningRate 

public double RotationLearningRate { get; set; }

Property Value

double

ScaleLearningRate 

public double ScaleLearningRate { get; set; }

Property Value

double

SpatialIndexRadius 

public int SpatialIndexRadius { get; set; }

Property Value

int

SplitGradientThreshold 

public double SplitGradientThreshold { get; set; }

Property Value

double

SplitOpacityFactor 

public double SplitOpacityFactor { get; set; }

Property Value

double

SplitOpacityMax 

public double SplitOpacityMax { get; set; }

Property Value

double

SplitPositionJitter 

public double SplitPositionJitter { get; set; }

Property Value

double

SplitScaleFactor 

public double SplitScaleFactor { get; set; }

Property Value

double

SupportsTraining

Indicates whether this network supports training (learning from data).

public override bool SupportsTraining { get; }

Property Value

bool

Remarks

For Beginners: Not all neural networks can learn. Some are designed only for making predictions with pre-set parameters. This property tells you if the network can learn from data.

TileSize 

public int TileSize { get; set; }

Property Value

int

Methods

Backpropagate(Tensor<T>)

Performs backpropagation to compute gradients for network parameters.

public override Tensor<T> Backpropagate(Tensor<T> outputGradient)

Parameters

outputGradient Tensor<T>

Returns

Tensor<T>

The gradients of the loss with respect to the network inputs.

Remarks

For Beginners: Backpropagation is how neural networks learn. After making a prediction, the network calculates how wrong it was (the error). Then it works backward through the layers to figure out how each parameter contributed to that error. This method handles that backward flow of information.

The "gradients" are numbers that tell us how to adjust each parameter to reduce the error.

API Change Note: The signature changed from Vector<T> to Tensor<T> to support multi-dimensional gradients. This is a breaking change. If you need backward compatibility, consider adding an overload that accepts Vector<T> and converts it internally to Tensor<T>.

Exceptions

InvalidOperationException

Thrown when the network is not in training mode or doesn't support training.

CreateNewInstance()

Creates a new instance of the same type as this neural network.

protected override IFullModel<T, Tensor<T>, Tensor<T>> CreateNewInstance()

Returns

IFullModel<T, Tensor<T>, Tensor<T>>

A new instance of the same neural network type.

Remarks

For Beginners: This creates a blank version of the same type of neural network.

It's used internally by methods like DeepCopy and Clone to create the right type of network before copying the data into it.

DeserializeNetworkSpecificData(BinaryReader)

Deserializes network-specific data that was not covered by the general deserialization process.

protected override void DeserializeNetworkSpecificData(BinaryReader reader)

Parameters

reader BinaryReader

The BinaryReader to read the data from.

Remarks

This method is called at the end of the general deserialization process to allow derived classes to read any additional data specific to their implementation.

For Beginners: Continuing the suitcase analogy, this is like unpacking that special compartment. After the main deserialization method has unpacked the common items (layers, parameters), this method allows each specific type of neural network to unpack its own unique items that were stored during serialization.

ForwardWithMemory(Tensor<T>)

Performs a forward pass through the network while storing intermediate values for backpropagation.

public override Tensor<T> ForwardWithMemory(Tensor<T> input)

Parameters

input Tensor<T>

The input data to the network.

Returns

Tensor<T>

The output of the network.

Remarks

For Beginners: This method passes data through the network from input to output, but also remembers all the intermediate values. This is necessary for the learning process, as the network needs to know these values when figuring out how to improve.

API Change Note: The signature changed from Vector<T> to Tensor<T> to support multi-dimensional inputs. This is a breaking change. For backward compatibility, consider adding an overload that accepts Vector<T> and converts it internally to Tensor<T>.

Exceptions

InvalidOperationException

Thrown when the network doesn't support training.

GetModelMetadata()

Gets the metadata for this neural network model.

public override ModelMetadata<T> GetModelMetadata()

Returns

ModelMetadata<T>

A ModelMetaData object containing information about the model.

InitializeLayers()

Initializes the layers of the neural network based on the architecture.

protected override void InitializeLayers()

Remarks

For Beginners: This method sets up all the layers in your neural network according to the architecture you've defined. It's like assembling the parts of your network before you can use it.

Predict(Tensor<T>)

Makes a prediction using the neural network.

public override Tensor<T> Predict(Tensor<T> input)

Parameters

input Tensor<T>

The input data to process.

Returns

Tensor<T>

The network's prediction.

Remarks

For Beginners: This is the main method you'll use to get results from your trained neural network. You provide some input data (like an image or text), and the network processes it through all its layers to produce an output (like a classification or prediction).

QueryField(Tensor<T>, Tensor<T>)

Queries the radiance field at specific 3D positions and viewing directions.

public (Tensor<T> rgb, Tensor<T> density) QueryField(Tensor<T> positions, Tensor<T> viewingDirections)

Parameters

positions Tensor<T>

Tensor of 3D positions [N, 3] where N is number of query points.

viewingDirections Tensor<T>

Tensor of viewing directions [N, 3] (unit vectors).

Returns

(Tensor<T> grad1, Tensor<T> grad2)

A tuple containing RGB colors [N, 3] and density values [N, 1].

Remarks

For Beginners: This is the core operation of a radiance field.

For each query point:

  • Position (x, y, z): Where in 3D space are we looking?
  • Direction (dx, dy, dz): Which direction are we looking from?

The network returns:

  • RGB (r, g, b): The color at that point from that direction
  • Density σ: How "solid" or "opaque" that point is

Example query:

  • Position: (2.5, 1.0, -3.0) - a point in space
  • Direction: (0.0, 0.0, -1.0) - looking straight down negative Z axis
  • Result: (Red: 0.8, Green: 0.3, Blue: 0.1, Density: 5.2) This means the point appears orange when viewed from that direction, and it's fairly opaque (high density)

Density interpretation:

  • Density = 0: Completely transparent (empty space)
  • Density > 0: Increasingly opaque (solid material)
  • Higher density = light is more likely to stop at this point

RenderImage(Vector<T>, Matrix<T>, int, int, T)

Renders an image from a specific camera position and orientation.

public Tensor<T> RenderImage(Vector<T> cameraPosition, Matrix<T> cameraRotation, int imageWidth, int imageHeight, T focalLength)

Parameters

cameraPosition Vector<T>

3D position of the camera [3].

cameraRotation Matrix<T>

Rotation matrix of the camera [3, 3].

imageWidth int

Width of the output image in pixels.

imageHeight int

Height of the output image in pixels.

focalLength T

Camera focal length.

Returns

Tensor<T>

Rendered RGB image tensor [height, width, 3].

Remarks

For Beginners: This renders a 2D image from the 3D scene representation.

The rendering process:

  1. For each pixel in the output image:
    • Cast a ray from camera through that pixel
    • Sample points along the ray
    • Query radiance field at each sample point
    • Combine colors using volume rendering (accumulate with alpha blending)

Volume rendering equation:

  • For each ray, accumulate: Color = Σ(transmittance × color × alpha)
  • Transmittance: How much light passes through previous points
  • Alpha: How much light is absorbed at this point (based on density)

Example:

  • Camera at (0, 0, 5) looking at origin
  • Image 512×512 pixels
  • Cast 512×512 = 262,144 rays
  • Sample 64 points per ray = 16.8 million queries
  • Blend results to get final image

This is why NeRF rendering can be slow - many network queries! Optimizations like Instant-NGP speed this up significantly.

RenderRays(Tensor<T>, Tensor<T>, int, T, T)

Renders rays through the radiance field using volume rendering.

public Tensor<T> RenderRays(Tensor<T> rayOrigins, Tensor<T> rayDirections, int numSamples, T nearBound, T farBound)

Parameters

rayOrigins Tensor<T>

Origins of rays to render [N, 3].

rayDirections Tensor<T>

Directions of rays (unit vectors) [N, 3].

numSamples int

Number of samples per ray.

nearBound T

Near clipping distance.

farBound T

Far clipping distance.

Returns

Tensor<T>

Rendered RGB colors for each ray [N, 3].

Remarks

For Beginners: Volume rendering is how we convert the radiance field into images.

For each ray:

  1. Sample points: Generate sample positions between near and far bounds
  2. Query field: Get RGB and density at each sample
  3. Compute alpha: Convert density to opacity for each segment
  4. Accumulate color: Blend colors front-to-back

The algorithm:

For each sample i along ray:
  alpha_i = 1 - exp(-density_i * distance_i)
  transmittance_i = exp(-sum of all previous densities)
  color_contribution_i = transmittance_i * alpha_i * color_i
  total_color += color_contribution_i

Example with 4 samples:

  • Sample 0: Empty space (density ≈ 0) → contributes little
  • Sample 1: Empty space (density ≈ 0) → contributes little
  • Sample 2: Surface (density high) → contributes most of the color
  • Sample 3: Behind surface → mostly blocked by sample 2

Parameters:

  • numSamples: More samples = better quality but slower (typical: 64-192)
  • nearBound/farBound: Define region to sample (e.g., 0.1 to 10.0 meters)

SerializeNetworkSpecificData(BinaryWriter)

Serializes network-specific data that is not covered by the general serialization process.

protected override void SerializeNetworkSpecificData(BinaryWriter writer)

Parameters

writer BinaryWriter

The BinaryWriter to write the data to.

Remarks

This method is called at the end of the general serialization process to allow derived classes to write any additional data specific to their implementation.

For Beginners: Think of this as packing a special compartment in your suitcase. While the main serialization method packs the common items (layers, parameters), this method allows each specific type of neural network to pack its own unique items that other networks might not have.

Train(Tensor<T>, Tensor<T>)

Trains the neural network on a single input-output pair.

public override void Train(Tensor<T> input, Tensor<T> expectedOutput)

Parameters

input Tensor<T>

The input data.

expectedOutput Tensor<T>

The expected output for the given input.

Remarks

This method performs one training step on the neural network using the provided input and expected output. It updates the network's parameters to reduce the error between the network's prediction and the expected output.

For Beginners: This is how your neural network learns. You provide: - An input (what the network should process) - The expected output (what the correct answer should be)

The network then:

  1. Makes a prediction based on the input
  2. Compares its prediction to the expected output
  3. Calculates how wrong it was (the loss)
  4. Adjusts its internal values to do better next time

After training, you can get the loss value using the GetLastLoss() method to see how well the network is learning.

UpdateParameters(Vector<T>)

Updates the network's parameters with new values.

public override void UpdateParameters(Vector<T> parameters)

Parameters

parameters Vector<T>

The new parameter values to set.

Remarks

For Beginners: During training, a neural network's internal values (parameters) get adjusted to improve its performance. This method allows you to update all those values at once by providing a complete set of new parameters.

This is typically used by optimization algorithms that calculate better parameter values based on training data.

Edit this page