Table of Contents

Class BayesianRegression<T>

Namespace
AiDotNet.Regression
Assembly
AiDotNet.dll

Implements Bayesian Linear Regression with support for various kernels and uncertainty estimation.

public class BayesianRegression<T> : RegressionBase<T>, IRegression<T>, IFullModel<T, Matrix<T>, Vector<T>>, IModel<Matrix<T>, Vector<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Matrix<T>, Vector<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Matrix<T>, Vector<T>>>, IGradientComputable<T, Matrix<T>, Vector<T>>, IJitCompilable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
BayesianRegression<T>
Implements
IFullModel<T, Matrix<T>, Vector<T>>
IModel<Matrix<T>, Vector<T>, ModelMetadata<T>>
IParameterizable<T, Matrix<T>, Vector<T>>
ICloneable<IFullModel<T, Matrix<T>, Vector<T>>>
IGradientComputable<T, Matrix<T>, Vector<T>>
Inherited Members
Extension Methods

Remarks

Bayesian Linear Regression extends traditional linear regression by using Bayesian inference to provide a probabilistic model of the regression problem. Instead of point estimates of the model parameters, it computes a full posterior distribution over the parameters, allowing for uncertainty quantification in predictions. The model assumes Gaussian prior distributions on the parameters and Gaussian noise in the observations.

This implementation supports various kernel functions for non-linear regression, including: - Linear kernel (standard linear regression) - Radial Basis Function (RBF) kernel - Polynomial kernel - Sigmoid kernel - Laplacian kernel The choice of kernel enables the model to capture different types of relationships between features and targets.

For Beginners: Bayesian regression is a special type of regression model that not only predicts values but also tells you how confident it is about those predictions.

Think of it this way: If you were to guess someone's weight just by looking at their height, you wouldn't be 100% sure about your guess. You'd have some uncertainty. Bayesian regression captures this uncertainty mathematically.

Key features of Bayesian regression:

  • It calculates probabilities instead of just point estimates
  • It can tell you which predictions are more reliable than others
  • It combines prior knowledge with observed data to make inferences
  • It can incorporate various "kernels" to model different types of relationships

A "kernel" is like a special lens that transforms how the model sees relationships in your data. For example, some kernels are good at capturing curved relationships, while others might be better for periodic patterns.

Bayesian regression is especially useful when:

  • You have limited data
  • You want to know how confident the model is in its predictions
  • You need to incorporate prior knowledge about the problem

Constructors

BayesianRegression(BayesianRegressionOptions<T>?, IRegularization<T, Matrix<T>, Vector<T>>?)

Initializes a new instance of the BayesianRegression<T> class with the specified options and regularization.

public BayesianRegression(BayesianRegressionOptions<T>? bayesianOptions = null, IRegularization<T, Matrix<T>, Vector<T>>? regularization = null)

Parameters

bayesianOptions BayesianRegressionOptions<T>

The options for configuring the Bayesian regression algorithm. If null, default options are used.

regularization IRegularization<T, Matrix<T>, Vector<T>>

Optional regularization to prevent overfitting.

Remarks

The constructor initializes the Bayesian regression model with the specified options and regularization. The options control parameters such as the prior precision (alpha), noise precision (beta), kernel type, and kernel-specific parameters.

For Beginners: This creates a new Bayesian regression model with specific settings.

The options parameter controls important settings like:

  • Alpha: Controls the strength of the prior belief about parameters (higher = stronger prior)
  • Beta: Controls the assumed noise level in the data (higher = less noise)
  • KernelType: Specifies what kind of relationship pattern to look for (linear, curved, etc.)
  • DecompositionType: Technical setting for how certain matrix operations are performed

The regularization parameter helps prevent "overfitting" - a situation where the model works well on training data but poorly on new data because it's too closely tailored to the specific examples it was trained on.

If you're not sure what values to use, the default options typically provide a good starting point for many regression problems.

Methods

CreateNewInstance()

Creates a new instance of the Bayesian regression model with the same configuration.

protected override IFullModel<T, Matrix<T>, Vector<T>> CreateNewInstance()

Returns

IFullModel<T, Matrix<T>, Vector<T>>

A new instance of BayesianRegression<T> with the same configuration as the current instance.

Remarks

This method creates a new Bayesian regression model that has the same configuration as the current instance. It's used for model persistence, cloning, and transferring the model's configuration to new instances.

For Beginners: This method makes a fresh copy of the current model with the same settings.

It's like creating a blueprint copy of your model that can be used to:

  • Save your model's settings
  • Create a new identical model
  • Transfer your model's configuration to another system

This is useful when you want to:

  • Create multiple similar models
  • Save a model's configuration for later use
  • Reset a model while keeping its settings

GetModelType()

Gets the model type for serialization purposes.

protected override ModelType GetModelType()

Returns

ModelType

The model type identifier.

Remarks

This method returns the type identifier for the Bayesian regression model, which is used during serialization and deserialization to correctly reconstruct the model.

For Beginners: This method simply identifies the type of model for saving and loading purposes.

When you save a model to a file or database, the system needs to know what kind of model it is in order to load it correctly later. This method provides that identification.

Predict(Matrix<T>)

Makes predictions on new data using the trained Bayesian regression model.

public override Vector<T> Predict(Matrix<T> input)

Parameters

input Matrix<T>

The input features matrix where each row is a sample to predict.

Returns

Vector<T>

The predicted values.

Remarks

This method performs predictions using the mean of the posterior distribution over the model parameters. The prediction process consists of the following steps: 1. Preprocess the input data (add intercept, apply kernel) 2. Compute the predicted values using the trained model parameters

For Beginners: This method uses the trained model to make predictions on new data.

Here's how the prediction works:

  1. The method first prepares your input data:

    • It adds a constant term if you're using an intercept
    • It applies the same kernel transformation used during training (if any)

  2. Then it multiplies the prepared input by the learned coefficients to get predictions

This method gives you the "expected" or "mean" prediction, without information about uncertainty. If you want uncertainty estimates as well, use the PredictWithUncertainty method instead.

PredictWithUncertainty(Matrix<T>)

Makes predictions with uncertainty estimates on new data using the trained Bayesian regression model.

public (Vector<T> Mean, Vector<T> Variance) PredictWithUncertainty(Matrix<T> input)

Parameters

input Matrix<T>

The input features matrix where each row is a sample to predict.

Returns

(Vector<T> mean, Vector<T> logVar)

A tuple containing the mean predictions and their variances.

Remarks

This method performs predictions using the full posterior distribution over the model parameters, providing both the mean prediction and the variance for each prediction. The variance represents the uncertainty in the prediction and is composed of two terms: 1. The uncertainty due to the model parameters (epistemic uncertainty) 2. The irreducible noise in the data (aleatoric uncertainty)

For Beginners: This method makes predictions AND tells you how confident the model is about each prediction.

For example, if predicting house prices:

  • A prediction of "$300,000 ± $10,000" is more confident than
  • A prediction of "$300,000 ± $50,000"

The method returns two values for each input:

  • Mean: The best guess prediction (same as the regular Predict method)
  • Variance: A measure of uncertainty or confidence in that prediction

This uncertainty comes from two sources:

  • Parameter uncertainty: How confident the model is about its learned coefficients
  • Noise uncertainty: The inherent randomness in the data that can't be explained

Having uncertainty estimates is extremely valuable for decision-making, risk assessment, and understanding when to trust or question the model's predictions.

Train(Matrix<T>, Vector<T>)

Trains the Bayesian regression model on the provided input data and target values.

public override void Train(Matrix<T> x, Vector<T> y)

Parameters

x Matrix<T>

The input features matrix where each row is a sample and each column is a feature.

y Vector<T>

The target values vector corresponding to the input samples.

Remarks

This method implements Bayesian inference for linear regression. It computes the posterior distribution over the regression coefficients given the input data, target values, and prior distribution. The main steps of the algorithm are: 1. Preprocess the input data (add intercept, apply kernel, regularize) 2. Compute the prior precision matrix (inverse of prior covariance) 3. Compute the data likelihood precision matrix 4. Compute the posterior precision matrix (prior + likelihood) 5. Invert the posterior precision to get the posterior covariance 6. Compute the posterior mean (coefficients)

For Beginners: This method teaches the model to make predictions based on your training data.

Here's what happens during training:

  1. The method first prepares your data:

    • It adds a constant term if you're using an intercept (like y-intercept in a line equation)
    • It applies a kernel transformation if you've selected a non-linear kernel
    • It applies regularization to help prevent overfitting

  2. Then it uses Bayesian math to:

    • Calculate how much to trust the prior beliefs (prior precision)
    • Calculate how much to trust the data (design precision)
    • Combine these to get the final model parameters
    • Store information about uncertainty for later use

Unlike regular regression that gives single "best" values for each coefficient, Bayesian regression captures a distribution of likely values, which allows it to estimate uncertainty in its predictions.

Edit this page