Class KullbackLeiblerDivergence<T>

Namespace

AiDotNet.LossFunctions

Assembly

AiDotNet.dll

Implements the Kullback-Leibler Divergence, a measure of how one probability distribution differs from another.

public class KullbackLeiblerDivergence<T> : LossFunctionBase<T>, ILossFunction<T>

Type Parameters

T

The numeric type used for calculations (e.g., float, double).

Inheritance

object

LossFunctionBase<T>

KullbackLeiblerDivergence<T>

Implements

ILossFunction<T>

Inherited Members

LossFunctionBase<T>.NumOps

LossFunctionBase<T>.Engine

LossFunctionBase<T>.CalculateLossAndGradientGpu(IGpuTensor<T>, IGpuTensor<T>)

LossFunctionBase<T>.ValidateVectorLengths(Vector<T>, Vector<T>)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

LossFunctionExtensions.ComputeGradient<T>(ILossFunction<T>, Tensor<T>, Tensor<T>)

LossFunctionExtensions.ComputeLoss<T>(ILossFunction<T>, Tensor<T>, Tensor<T>)

Remarks

For Beginners: Kullback-Leibler (KL) Divergence measures how one probability distribution differs from another. It's often interpreted as the "information loss" when using one distribution to approximate another.

The formula is: KL(P||Q) = sum(P(i) * log(P(i)/Q(i)) Where:

• P is the true distribution
• Q is the approximating distribution

Key properties:

• It's always non-negative (zero only when the distributions are identical)
• It's not symmetric: KL(P||Q) ? KL(Q||P)
• It's not a true distance metric due to this asymmetry

KL divergence is commonly used in:

• Variational Autoencoders (VAEs)
• Reinforcement learning algorithms
• Information theory applications
• Distribution approximation tasks

When training models, KL divergence helps push the predicted distribution (Q) to match the target distribution (P).

Constructors

KullbackLeiblerDivergence()

Initializes a new instance of the KullbackLeiblerDivergence class.

public KullbackLeiblerDivergence()

Methods

CalculateDerivative(Vector<T>, Vector<T>)

Calculates the derivative of the Kullback-Leibler Divergence.

public override Vector<T> CalculateDerivative(Vector<T> predicted, Vector<T> actual)

Parameters

predicted Vector<T>

The predicted probability distribution.

actual Vector<T>

The actual (target) probability distribution.

Returns

Vector<T>

A vector containing the gradient of the KL divergence with respect to each prediction.

CalculateLoss(Vector<T>, Vector<T>)

Calculates the Kullback-Leibler Divergence between predicted and actual probability distributions.

public override T CalculateLoss(Vector<T> predicted, Vector<T> actual)

Parameters

predicted Vector<T>

The predicted probability distribution.

actual Vector<T>

The actual (target) probability distribution.

Returns

T

The KL divergence value.