Table of Contents

Class GradientBoostingRegression<T>

Namespace
AiDotNet.Regression
Assembly
AiDotNet.dll

Implements a Gradient Boosting Regression model, which combines multiple decision trees sequentially to create a powerful ensemble that learns from the errors of previous trees.

public class GradientBoostingRegression<T> : AsyncDecisionTreeRegressionBase<T>, IAsyncTreeBasedModel<T>, ITreeBasedRegression<T>, INonLinearRegression<T>, IRegression<T>, IFullModel<T, Matrix<T>, Vector<T>>, IModel<Matrix<T>, Vector<T>, ModelMetadata<T>>, IModelSerializer, ICheckpointableModel, IParameterizable<T, Matrix<T>, Vector<T>>, IFeatureAware, IFeatureImportance<T>, ICloneable<IFullModel<T, Matrix<T>, Vector<T>>>, IGradientComputable<T, Matrix<T>, Vector<T>>, IJitCompilable<T>

Type Parameters

T

The numeric type used for calculations, typically float or double.

Inheritance
GradientBoostingRegression<T>
Implements
IFullModel<T, Matrix<T>, Vector<T>>
IModel<Matrix<T>, Vector<T>, ModelMetadata<T>>
IParameterizable<T, Matrix<T>, Vector<T>>
ICloneable<IFullModel<T, Matrix<T>, Vector<T>>>
IGradientComputable<T, Matrix<T>, Vector<T>>
Inherited Members
Extension Methods

Remarks

Gradient Boosting is an ensemble technique that builds decision trees sequentially, with each tree correcting the errors made by the previous trees. The model starts with a simple prediction (typically the mean of the target values) and iteratively adds trees that predict the residuals (errors) of the current ensemble. These predictions are added to the ensemble with a learning rate that controls the contribution of each tree, helping to prevent overfitting.

For Beginners: Gradient Boosting is like having a team of experts who learn from each other's mistakes.

Imagine you're trying to predict house prices:

  • You start with a simple guess (the average price of all houses)
  • You build a decision tree to predict where your guess was wrong
  • You adjust your prediction a little bit based on this tree
  • You build another tree to predict where you're still making mistakes
  • You keep adding trees, each one focusing on fixing the remaining errors

The "gradient" part refers to how it identifies mistakes, and "boosting" means it builds trees sequentially, with each tree boosting the performance of the ensemble.

This approach is very powerful because:

  • It learns complex patterns gradually
  • It focuses its effort on the hard-to-predict cases
  • It combines many simple models (trees) into a strong predictive model

Constructors

GradientBoostingRegression(GradientBoostingRegressionOptions?, IRegularization<T, Matrix<T>, Vector<T>>?)

Initializes a new instance of the GradientBoostingRegression<T> class.

public GradientBoostingRegression(GradientBoostingRegressionOptions? options = null, IRegularization<T, Matrix<T>, Vector<T>>? regularization = null)

Parameters

options GradientBoostingRegressionOptions

Optional configuration options for the Gradient Boosting algorithm.

regularization IRegularization<T, Matrix<T>, Vector<T>>

Optional regularization strategy to prevent overfitting.

Remarks

This constructor creates a new Gradient Boosting Regression model with the specified options and regularization strategy. If no options are provided, default values are used. If no regularization is specified, no regularization is applied.

For Beginners: This is how you create a new Gradient Boosting model.

When creating a model, you can specify:

  • Options: Controls how many trees to build, how complex each tree can be, and how quickly the model learns
  • Regularization: Helps prevent the model from becoming too specialized to the training data

If you don't specify these parameters, the model will use reasonable default settings.

Example:

// Create a Gradient Boosting model with default settings
var gbr = new GradientBoostingRegression<double>();

// Create a model with custom options
var options = new GradientBoostingRegressionOptions { 
    NumberOfTrees = 100,
    LearningRate = 0.1,
    MaxDepth = 3
};
var customGbr = new GradientBoostingRegression<double>(options);

Properties

NumberOfTrees

Gets the number of trees in the ensemble model.

public override int NumberOfTrees { get; }

Property Value

int

The number of trees in the ensemble.

Remarks

This property returns the number of decision trees in the Gradient Boosting ensemble. This is an important characteristic of the model as it represents the number of boosting stages that have been performed.

For Beginners: This tells you how many individual decision trees are in your model.

In Gradient Boosting:

  • Each tree corrects errors made by all previous trees
  • More trees generally means better predictions (up to a point)
  • However, too many trees can lead to overfitting

Typical gradient boosting models might use anywhere from 50 to 1000 trees, depending on the complexity of the problem and the depth of each tree.

SupportsJitCompilation

Gets whether this Gradient Boosting model supports JIT compilation.

public override bool SupportsJitCompilation { get; }

Property Value

bool

true when soft tree mode is enabled and trees have been trained; false otherwise.

Remarks

Gradient Boosting supports JIT compilation when soft tree mode is enabled. In soft mode, each tree in the ensemble uses sigmoid-based soft gating instead of hard if-then splits, making the entire sequential ensemble differentiable.

The computation graph follows the gradient boosting formula: 

prediction = initial_prediction + learning_rate × Σ tree_i(input)

For Beginners: JIT compilation is available when soft tree mode is enabled.

In soft tree mode:

  • Each tree in the boosted ensemble uses smooth transitions
  • The sequential ensemble can be exported as a single computation graph
  • The learning rate and initial prediction are embedded in the graph

This gives you the benefits of gradient boosting with JIT-compiled speed.

Methods

CalculateFeatureImportancesAsync(int)

Calculates the importance scores for all features used in the model.

protected override Task CalculateFeatureImportancesAsync(int featureCount)

Parameters

featureCount int

The number of features in the model.

Returns

Task

A task representing the asynchronous calculation operation.

Remarks

This method calculates the importance of each feature in the Gradient Boosting model by aggregating the importance scores across all trees in the ensemble. The importance scores are normalized to sum to 1, making it easier to compare the relative importance of different features.

For Beginners: This method figures out which input features matter most for predictions.

Feature importance helps you understand:

  • Which variables have the biggest impact on your predictions
  • Which features might be redundant or irrelevant
  • What the model is focusing on when making decisions

The calculation works by:

  1. Getting the importance scores from each individual tree
  2. Adding up these scores across all trees for each feature
  3. Normalizing the scores so they sum to 1 (making them easier to compare)

This information can help you interpret the model and potentially simplify future models by focusing on the most important features.

CreateNewInstance()

Creates a new instance of the gradient boosting regression model with the same configuration.

protected override IFullModel<T, Matrix<T>, Vector<T>> CreateNewInstance()

Returns

IFullModel<T, Matrix<T>, Vector<T>>

A new instance of GradientBoostingRegression<T> with the same configuration as the current instance.

Remarks

This method creates a new gradient boosting regression model that has the same configuration as the current instance. It's used for model persistence, cloning, and transferring the model's configuration to new instances.

For Beginners: This method makes a fresh copy of the current model with the same settings.

It's like creating a blueprint copy of your model that can be used to:

  • Save your model's settings
  • Create a new identical model
  • Transfer your model's configuration to another system

This is useful when you want to:

  • Create multiple similar models
  • Save a model's configuration for later use
  • Reset a model while keeping its settings

Deserialize(byte[])

Loads a previously serialized Gradient Boosting Regression model from a byte array.

public override void Deserialize(byte[] modelData)

Parameters

modelData byte[]

The byte array containing the serialized model.

Remarks

This method reconstructs a Gradient Boosting Regression model from a byte array that was previously created using the Serialize method. It restores the base class data, model-specific options, the initial prediction, and all the trees in the ensemble, allowing the model to be used for predictions without retraining.

For Beginners: This method loads a previously saved model from a sequence of bytes.

Deserialization allows you to:

  • Load a model that was saved earlier
  • Use a model without having to retrain it
  • Share models between different applications

When you deserialize a model:

  • All settings are restored
  • The initial prediction is recovered
  • All the individual trees are reconstructed
  • The model is ready to make predictions immediately

Example:

// Load from a file
byte[] modelData = File.ReadAllBytes("gradientBoosting.model");

// Deserialize the model
var gbr = new GradientBoostingRegression<double>();
gbr.Deserialize(modelData);

// Now you can use the model for predictions
var predictions = await gbr.PredictAsync(newFeatures);

ExportComputationGraph(List<ComputationNode<T>>)

Exports the Gradient Boosting model's computation graph for JIT compilation.

public override ComputationNode<T> ExportComputationGraph(List<ComputationNode<T>> inputNodes)

Parameters

inputNodes List<ComputationNode<T>>

List to populate with input computation nodes.

Returns

ComputationNode<T>

The root node of the exported computation graph.

Remarks

When soft tree mode is enabled, this exports the entire Gradient Boosting ensemble as a differentiable computation graph. The graph follows the formula: 

output = initial_prediction + learning_rate × (tree1 + tree2 + ... + treeN)
where each tree uses soft split operations.

For Beginners: This exports the gradient boosted ensemble as a computation graph.

Unlike Random Forest (which averages tree outputs), Gradient Boosting:

  • Starts with an initial prediction (mean of training targets)
  • Adds contributions from each tree scaled by the learning rate
  • Each tree predicts "residuals" (errors from previous trees)

The exported graph combines all these elements into optimized code.

Exceptions

NotSupportedException

Thrown when soft tree mode is not enabled.

InvalidOperationException

Thrown when the model has not been trained (no trees).

GetModelMetadata()

Gets metadata about the Gradient Boosting Regression model and its configuration.

public override ModelMetadata<T> GetModelMetadata()

Returns

ModelMetadata<T>

A ModelMetadata object containing information about the model.

Remarks

This method returns metadata about the model, including its type and configuration options such as the number of trees, maximum tree depth, learning rate, and subsampling ratio. This information can be useful for model management, comparison, and documentation purposes.

For Beginners: This method provides information about your Gradient Boosting model.

The metadata includes:

  • The type of model (Gradient Boosting)
  • How many trees are in the ensemble
  • How deep each tree is allowed to grow
  • The learning rate (how quickly the model incorporates new trees)
  • Subsampling ratio (what fraction of the data is used for each tree)
  • Other configuration settings

This information is helpful when:

  • Comparing different models
  • Documenting your model's configuration
  • Troubleshooting model performance
  • Replicating your results

Example:

var metadata = gbr.GetModelMetadata();
Console.WriteLine($"Model type: {metadata.ModelType}");
Console.WriteLine($"Number of trees: {metadata.AdditionalInfo["NumberOfTrees"]}");
Console.WriteLine($"Learning rate: {metadata.AdditionalInfo["LearningRate"]}");

PredictAsync(Matrix<T>)

Asynchronously predicts target values for the provided input features using the trained Gradient Boosting model.

public override Task<Vector<T>> PredictAsync(Matrix<T> input)

Parameters

input Matrix<T>

A matrix where each row represents a sample to predict and each column represents a feature.

Returns

Task<Vector<T>>

A task that returns a vector of predicted values corresponding to each input sample.

Remarks

This method predicts target values for new input data by combining the initial prediction with the weighted contributions of all trees in the ensemble. The predictions from each tree are scaled by the learning rate before being added to the ensemble prediction. The method uses parallel processing to generate tree predictions efficiently.

For Beginners: This method uses your trained model to make predictions on new data.

The prediction process works like this:

  1. Start with the initial prediction (the average of all target values in the training data)
  2. For each tree in the model:
    • Get the tree's prediction
    • Scale it by the learning rate (to control how much influence each tree has)
    • Add it to the running total
  3. The final prediction is the sum of the initial prediction plus all the scaled tree predictions

The "Async" in the name means this method returns a Task, allowing your program to do other things while waiting for predictions to complete. It also uses parallel processing to get predictions from multiple trees simultaneously, making it faster.

Example:

// Make predictions
var predictions = await gbr.PredictAsync(newFeatures);

Serialize()

Serializes the Gradient Boosting Regression model to a byte array for storage or transmission.

public override byte[] Serialize()

Returns

byte[]

A byte array containing the serialized model.

Remarks

This method converts the Gradient Boosting Regression model into a byte array that can be stored in a file, database, or transmitted over a network. The serialized data includes the base class data, model-specific options, the initial prediction, and all the trees in the ensemble.

For Beginners: This method saves your trained model as a sequence of bytes.

Serialization allows you to:

  • Save your model to a file
  • Store your model in a database
  • Send your model over a network
  • Keep your model for later use without having to retrain it

The serialized data includes:

  • All the model's settings (like number of trees and learning rate)
  • The initial prediction (the starting point for all predictions)
  • Every individual decision tree in the ensemble

Because Gradient Boosting models contain multiple trees, the serialized data can be quite large for complex models.

Example:

// Serialize the model
byte[] modelData = gbr.Serialize();

// Save to a file
File.WriteAllBytes("gradientBoosting.model", modelData);

TrainAsync(Matrix<T>, Vector<T>)

Asynchronously trains the Gradient Boosting Regression model using the provided input features and target values.

public override Task TrainAsync(Matrix<T> x, Vector<T> y)

Parameters

x Matrix<T>

A matrix where each row represents a sample and each column represents a feature.

y Vector<T>

A vector of target values corresponding to each sample in x.

Returns

Task

A task representing the asynchronous training operation.

Remarks

This method trains the Gradient Boosting Regression model by first calculating an initial prediction (typically the mean of the target values), and then sequentially building decision trees that predict the residuals (errors) of the current ensemble. The trees are built in parallel to improve training efficiency, but the sequential nature of the algorithm is maintained by updating the residuals after each tree is built.

For Beginners: This method teaches the model how to make predictions using your data.

The training process works like this:

  1. Start with a simple prediction (the average of all target values)
  2. Calculate how wrong this prediction is for each training example (the "residuals")
  3. Build a decision tree that tries to predict these residuals
  4. Add this tree's predictions (scaled by the learning rate) to the current model
  5. Update the residuals based on the new predictions
  6. Repeat steps 3-5 until you've built the desired number of trees

The "Async" in the name means this method can run without blocking other operations in your program, and it uses parallel processing to build trees more quickly when possible.

Example:

// Train the model
await gbr.TrainAsync(features, targets);
Edit this page