Class NoiseContrastiveEstimationLoss<T>

Namespace

AiDotNet.LossFunctions

Assembly

AiDotNet.dll

Implements the Noise Contrastive Estimation (NCE) loss function for efficient training with large output spaces.

public class NoiseContrastiveEstimationLoss<T> : LossFunctionBase<T>, ILossFunction<T>

Type Parameters

T

The numeric type used for calculations (e.g., float, double).

Inheritance

object

LossFunctionBase<T>

NoiseContrastiveEstimationLoss<T>

Implements

ILossFunction<T>

Inherited Members

LossFunctionBase<T>.NumOps

LossFunctionBase<T>.Engine

LossFunctionBase<T>.CalculateLossAndGradientGpu(IGpuTensor<T>, IGpuTensor<T>)

LossFunctionBase<T>.ValidateVectorLengths(Vector<T>, Vector<T>)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

LossFunctionExtensions.ComputeGradient<T>(ILossFunction<T>, Tensor<T>, Tensor<T>)

LossFunctionExtensions.ComputeLoss<T>(ILossFunction<T>, Tensor<T>, Tensor<T>)

Remarks

For Beginners: Noise Contrastive Estimation (NCE) is a loss function designed to efficiently train models with very large output spaces, such as language models with large vocabularies.

Instead of computing probabilities for all possible outputs (which could be millions in language models), NCE transforms the problem into a binary classification task: distinguishing the true data from noise samples.

The key idea is to:

• Sample a small number of "negative" examples from a noise distribution
• Train the model to distinguish between true data points and these negative samples

This approach is much more computationally efficient than computing full softmax probabilities over all possible outputs, especially when the output space is very large.

NCE is commonly used in:

• Word embedding models like Word2Vec
• Neural language models with large vocabularies
• Any model with a very large output space

Constructors

NoiseContrastiveEstimationLoss(int)

Initializes a new instance of the NoiseContrastiveEstimationLoss class.

public NoiseContrastiveEstimationLoss(int numNoiseSamples = 10)

Parameters

numNoiseSamples int

The number of noise samples to use per true sample. Default is 10.

Methods

Calculate(Vector<T>, Matrix<T>)

Calculates the NCE loss between target and noise samples.

public T Calculate(Vector<T> targetLogits, Matrix<T> noiseLogits)

Parameters

targetLogits Vector<T>

The logits for the true target samples.

noiseLogits Matrix<T>

The logits for the noise samples.

Returns

T

The NCE loss value.

CalculateDerivative(Vector<T>, Matrix<T>)

Calculates the gradient of the NCE loss function.

public (Vector<T>, Matrix<T>) CalculateDerivative(Vector<T> targetLogits, Matrix<T> noiseLogits)

Parameters

targetLogits Vector<T>

The logits for the true target samples.

noiseLogits Matrix<T>

The logits for the noise samples.

Returns

(Vector<T> means, Matrix<T> covariance)

A tuple containing the gradients for target and noise logits.

CalculateDerivative(Vector<T>, Vector<T>)

This method is not used for NCE Loss as it requires specific input formats.

public override Vector<T> CalculateDerivative(Vector<T> predicted, Vector<T> actual)

Parameters

predicted Vector<T>

The predicted values vector.

actual Vector<T>

The actual (target) values vector.

Returns

Vector<T>

Throws NotSupportedException.

Exceptions

NotSupportedException

Always thrown as NCE Loss requires specific input formats.

CalculateLoss(Vector<T>, Vector<T>)

This method is not used for NCE Loss as it requires specific input formats.

public override T CalculateLoss(Vector<T> predicted, Vector<T> actual)

Parameters

predicted Vector<T>

The predicted values vector.

actual Vector<T>

The actual (target) values vector.

Returns

T

Throws NotSupportedException.

Exceptions

NotSupportedException

Always thrown as NCE Loss requires specific input formats.