Class ErrorStats<T>

Namespace

AiDotNet.Statistics

Assembly

AiDotNet.dll

Calculates and stores various error metrics for evaluating prediction model performance.

public class ErrorStats<T>

Type Parameters

T

The numeric type used for calculations (e.g., float, double, decimal).

Inheritance

object

ErrorStats<T>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This class provides a comprehensive set of error metrics to assess how well predicted values match actual values.

For Beginners: When building AI or machine learning models, you need ways to measure how accurate your predictions are. Think of these metrics like different ways to score a test:

• Some look at the average error (MAE, MSE)
• Some look at percentage differences (MAPE, SMAPE)
• Some help detect if your model is consistently overestimating or underestimating (MeanBiasError)
• Some are specialized for specific types of predictions (AUCROC, AUCPR for classification)

The "T" in ErrorStats<T> means this class works with different number types like decimal, double, or float without needing separate implementations for each.

Properties

AIC

Akaike Information Criterion - A measure that balances model accuracy and complexity.

public T AIC { get; }

Property Value

T

Remarks

For Beginners: AIC helps you compare different models by considering both how well they fit the data and how complex they are. Lower values are better.

Think of it like buying a car: You want good performance (accuracy) but don't want to pay too much (complexity). AIC helps you find the best balance.

AICAlt

Alternative Akaike Information Criterion - A variant of AIC with a different penalty term.

public T AICAlt { get; }

Property Value

T

Remarks

For Beginners: AICAlt is another version of AIC that uses a slightly different approach to penalize model complexity. It's particularly useful when sample sizes are small. Like AIC and BIC, lower values are better.

AUC

Area Under the Curve (ROC) - Alias for AUCROC property.

public T AUC { get; }

Property Value

T

Remarks

For Beginners: This is an alternative name for the AUCROC property, providing the same value. In many contexts, "AUC" specifically refers to the area under the ROC curve. This metric is commonly used to evaluate classification models.

AUCPR

Area Under the Precision-Recall Curve - Measures classification accuracy focusing on positive cases.

public T AUCPR { get; }

Property Value

T

Remarks

For Beginners: AUCPR is especially useful for imbalanced classification problems (where one class is rare). It ranges from 0 to 1, with higher values indicating better performance.

Precision measures how many of your positive predictions were correct. Recall measures what fraction of actual positives your model identified. AUCPR considers how these trade off across different threshold settings.

AUCROC

Area Under the Receiver Operating Characteristic Curve - Measures classification accuracy across thresholds.

public T AUCROC { get; }

Property Value

T

Remarks

For Beginners: AUCROC is a common metric for classification models. It ranges from 0 to 1:

• 0.5 means the model is no better than random guessing
• 1.0 means perfect classification
• Values below 0.5 suggest the model is worse than random

It measures how well your model can distinguish between classes across different threshold settings.

Accuracy

Classification accuracy - The proportion of correct predictions (for classification tasks).

public T Accuracy { get; }

Property Value

T

Remarks

For Beginners: Accuracy is a simple metric for classification problems. It's the percentage of predictions that match the actual values.

For example, if your model correctly classifies 90 out of 100 samples, the accuracy is 0.9 or 90%.

Note: This property is typically used for classification tasks. For regression tasks, other metrics like MAE, MSE, or R-squared (R2) are more appropriate.

While intuitive, accuracy can be misleading for imbalanced classes. For example, if 95% of your data belongs to class A, a model that always predicts class A would have 95% accuracy despite being useless for class B.

BIC

Bayesian Information Criterion - Similar to AIC but penalizes model complexity more strongly.

public T BIC { get; }

Property Value

T

Remarks

For Beginners: BIC is similar to AIC but tends to prefer simpler models. It helps you compare different models while avoiding overfitting (when a model memorizes training data instead of learning general patterns). Lower values are better.

BIC is more cautious about adding complexity than AIC, like a budget-conscious car buyer.

CRPS

Continuous Ranked Probability Score - Evaluates probabilistic forecast accuracy.

public T CRPS { get; }

Property Value

T

Remarks

For Beginners: CRPS measures how well a probabilistic forecast matches reality. Unlike MAE which only looks at point predictions, CRPS considers the entire predicted probability distribution.

For deterministic predictions (no uncertainty estimates), CRPS equals MAE. For probabilistic predictions, CRPS rewards well-calibrated uncertainty estimates. Lower values indicate better probabilistic forecasts.

CRPS is especially important for evaluating time series models like DeepAR, TFT, and Chronos that provide uncertainty estimates along with their predictions.

DurbinWatsonStatistic

Durbin-Watson Statistic - Detects autocorrelation in prediction errors.

public T DurbinWatsonStatistic { get; }

Property Value

T

Remarks

For Beginners: DurbinWatsonStatistic helps identify if there are patterns in your prediction errors over time. Values range from 0 to 4:

• Values near 2 suggest no autocorrelation (good)
• Values toward 0 suggest positive autocorrelation (errors tend to be followed by similar errors)
• Values toward 4 suggest negative autocorrelation (errors tend to be followed by opposite errors)

Autocorrelation in errors suggests your model might be missing important patterns in the data.

ErrorList

List of individual prediction errors (residuals).

public List<T> ErrorList { get; }

Property Value

List<T>

Remarks

For Beginners: ErrorList contains the difference between each predicted value and the corresponding actual value. This lets you examine individual errors, create visualizations like histograms, or perform additional analyses beyond summary statistics.

F1Score

The harmonic mean of precision and recall (for classification).

public T F1Score { get; }

Property Value

T

Remarks

For Beginners: F1Score balances precision and recall in a single metric, which is helpful because there's often a trade-off between them.

It ranges from 0 to 1, with 1 being perfect.

F1Score is particularly useful when:

• You need a single metric to compare models
• Classes are imbalanced (one class is much more common than others)
• You care equally about false positives and false negatives

It's calculated as 2 * (precision * recall) / (precision + recall).

MAE

Mean Absolute Error - The average absolute difference between predicted and actual values.

public T MAE { get; }

Property Value

T

Remarks

For Beginners: MAE measures the average size of errors without considering their direction (positive or negative). Lower values indicate better accuracy. If MAE = 5, your predictions are off by 5 units on average.

MAPE

Mean Absolute Percentage Error - The average percentage difference between predicted and actual values.

public T MAPE { get; }

Property Value

T

Remarks

For Beginners: MAPE expresses error as a percentage, which helps you understand the relative size of errors. MAPE = 10 means that, on average, your predictions are off by 10% from the actual values. Note: MAPE can be problematic when actual values are close to zero.

MSE

Mean Squared Error - The average of squared differences between predicted and actual values.

public T MSE { get; }

Property Value

T

Remarks

For Beginners: MSE squares the errors before averaging them, which penalizes large errors more heavily than small ones. Lower values indicate better accuracy. Because of squaring, the value is not in the same units as your data.

MaxError

Maximum Error - The largest absolute difference between any predicted and actual value.

public T MaxError { get; }

Property Value

T

Remarks

For Beginners: MaxError tells you the worst-case error in your predictions. It can help you understand the potential maximum impact of prediction errors.

MeanAbsoluteError

Mean Absolute Error - Alias for MAE property.

public T MeanAbsoluteError { get; }

Property Value

T

Remarks

For Beginners: This is an alternative name for the MAE property, providing the same value. Some frameworks and documentation prefer the full name "MeanAbsoluteError" while others use "MAE". Both refer to the average absolute difference between predicted and actual values.

MeanBiasError

Mean Bias Error - The average of prediction errors (predicted - actual).

public T MeanBiasError { get; }

Property Value

T

Remarks

For Beginners: MeanBiasError helps determine if your model tends to overestimate (positive value) or underestimate (negative value). Ideally, it should be close to zero, indicating no systematic bias. Unlike MAE, this doesn't take the absolute value, so positive and negative errors can cancel out.

MeanSquaredError

Mean Squared Error - Alias for MSE property.

public T MeanSquaredError { get; }

Property Value

T

Remarks

For Beginners: This is an alternative name for the MSE property, providing the same value. Some frameworks and documentation prefer the full name "MeanSquaredError" while others use "MSE". Both refer to the average of squared differences between predicted and actual values.

MeanSquaredLogError

Mean Squared Logarithmic Error - Penalizes underestimates more than overestimates.

public T MeanSquaredLogError { get; }

Property Value

T

Remarks

For Beginners: MeanSquaredLogError is useful when you care more about relative errors than absolute ones. It's calculated by applying logarithms to actual and predicted values before computing MSE.

MSLE penalizes underestimation (predicting too low) more heavily than overestimation. This is useful in scenarios where underestimating would be more problematic, like inventory forecasting.

MedianAbsoluteError

Median Absolute Error - The middle value of all absolute differences between predicted and actual values.

public T MedianAbsoluteError { get; }

Property Value

T

Remarks

For Beginners: Unlike MAE which uses the average, MedianAbsoluteError uses the middle value of all absolute errors. This makes it less sensitive to outliers (extreme errors) than MAE. For example, if you have errors of [1, 2, 100], the median is 2, while the mean would be 34.3.

PopulationStandardError

Population Standard Error - The standard deviation of prediction errors without adjustment for model complexity.

public T PopulationStandardError { get; }

Property Value

T

Remarks

For Beginners: PopulationStandardError measures how much prediction errors typically vary, but unlike SampleStandardError, it doesn't adjust for model complexity. It gives you an idea of the typical size of the errors your model makes.

Precision

The proportion of positive predictions that were actually correct (for classification).

public T Precision { get; }

Property Value

T

Remarks

For Beginners: Precision answers the question: "Of all the items labeled as positive, how many actually were positive?"

It ranges from 0 to 1, with 1 being perfect.

For example, if your model identifies 100 emails as spam, and 90 of them actually are spam, the precision is 0.9 or 90%.

Precision is important when the cost of false positives is high. In the spam example, high precision means fewer important emails mistakenly marked as spam.

RMSE

Root Mean Squared Error - The square root of the Mean Squared Error.

public T RMSE { get; }

Property Value

T

Remarks

For Beginners: RMSE converts MSE back to the original units of your data by taking the square root. It's often preferred over MSE for interpretation because it's in the same units as your data. Like MAE, lower values indicate better accuracy.

RSS

Residual Sum of Squares - The sum of squared differences between predicted and actual values.

public T RSS { get; }

Property Value

T

Remarks

For Beginners: RSS is the total squared error of your model. It's the basis for many other metrics like MSE (which is just RSS divided by the number of observations). Lower values indicate a better fit. It's used in calculating metrics like AIC and BIC.

Recall

The proportion of actual positive cases that were correctly identified (for classification).

public T Recall { get; }

Property Value

T

Remarks

For Beginners: Recall answers the question: "Of all the actual positive items, how many did the model identify?"

It ranges from 0 to 1, with 1 being perfect.

For example, if there are 100 spam emails, and your model identifies 80 of them, the recall is 0.8 or 80%.

Recall is important when the cost of false negatives is high. In a medical context, high recall means catching most cases of a disease, even if it means some false alarms.

RootMeanSquaredError

Root Mean Squared Error - Alias for RMSE property.

public T RootMeanSquaredError { get; }

Property Value

T

Remarks

For Beginners: This is an alternative name for the RMSE property, providing the same value. Some frameworks and documentation prefer the full name "RootMeanSquaredError" while others use "RMSE". Both refer to the square root of the Mean Squared Error.

SMAPE

Symmetric Mean Absolute Percentage Error - A variant of MAPE that handles zero or near-zero values better.

public T SMAPE { get; }

Property Value

T

Remarks

For Beginners: SMAPE is similar to MAPE but uses a different formula that handles cases where actual values are zero or very small. It's bounded between 0% and 200%, with lower values indicating better performance.

SMAPE treats positive and negative errors more symmetrically than MAPE, which can be important in some forecasting applications.

SampleStandardError

Sample Standard Error - An estimate of the standard deviation of prediction errors, adjusted for model complexity.

public T SampleStandardError { get; }

Property Value

T

Remarks

For Beginners: SampleStandardError estimates how much prediction errors typically vary, taking into account how many parameters (features) your model uses. It's useful for constructing confidence intervals around predictions and is adjusted downward based on the number of parameters in your model.

TheilUStatistic

Theil's U Statistic - A measure of forecast accuracy relative to a naive forecasting method.

public T TheilUStatistic { get; }

Property Value

T

Remarks

For Beginners: TheilUStatistic compares your model's accuracy to a simple "no-change" prediction.

• Values less than 1 mean your model is better than the naive approach.
• Values equal to 1 mean your model performs the same as the naive approach.
• Values greater than 1 mean your model performs worse than the naive approach.

This is especially useful for time series forecasting evaluation.

Methods

Empty()

Creates an empty ErrorStats instance with all metrics set to zero.

public static ErrorStats<T> Empty()

Returns

ErrorStats<T>

An ErrorStats instance with all metrics initialized to zero.

Remarks

For Beginners: This static method creates an ErrorStats object where all metrics are set to zero. It's useful when you need a placeholder or default instance, or when you want to compare against a baseline of "no errors."

GetMetric(MetricType)

Retrieves the value of a specific error metric.

public T GetMetric(MetricType metricType)

Parameters

metricType MetricType

The type of metric to retrieve.

Returns

T

The value of the requested metric.

Remarks

This method allows you to retrieve any of the calculated error metrics by specifying the desired metric type. It provides a flexible way to access individual metrics without needing to reference specific properties.

For Beginners: This method is like a vending machine for error metrics.

You tell it which error metric you want (using the MetricType), and it gives you the value. For example:

• If you ask for MetricType.MAE, it gives you the Mean Absolute Error
• If you ask for MetricType.RMSE, it gives you the Root Mean Squared Error

This is useful when you want to work with different error metrics in a flexible way, especially if you don't know in advance which metric you'll need.

Exceptions

ArgumentException

Thrown when an unsupported MetricType is provided.

HasMetric(MetricType)

Checks if a specific metric is available in this ErrorStats instance.

public bool HasMetric(MetricType metricType)

Parameters

metricType MetricType

The type of metric to check for.

Returns

bool

True if the metric is available, false otherwise.

Remarks

For Beginners: This method allows you to check if a particular metric is available before trying to get its value. It's useful when you're not sure if a specific metric was calculated for this set of errors.

For example:

if (stats.HasMetric(MetricType.MAE))
{
    var maeValue = stats.GetMetric(MetricType.MAE);
    // Use maeValue...
}

This prevents errors that might occur if you try to access a metric that wasn't calculated.