Class FineTuningMetrics<T>

Namespace

AiDotNet.Models.Options

Assembly

AiDotNet.dll

Metrics for evaluating fine-tuning quality.

public class FineTuningMetrics<T>

Type Parameters

T

The numeric data type.

Inheritance

object

FineTuningMetrics<T>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This class contains metrics relevant to various fine-tuning methods:

• Loss metricsTraining and validation loss
• Preference metricsWin rate, preference accuracy
• RL metricsReward scores, KL divergence
• Safety metricsHarmlessness, refusal rates

For Beginners: These metrics tell you how well the fine-tuning worked. Lower loss is generally better, higher win rates mean the model learned preferences well, and safety metrics ensure the model behaves appropriately.

Properties

AverageOutputLength

Gets or sets the average output length.

public double AverageOutputLength { get; set; }

Property Value

double

AverageReward

Gets or sets the average reward achieved.

public double AverageReward { get; set; }

Property Value

double

BleuScore

Gets or sets the BLEU score for generation quality.

public double BleuScore { get; set; }

Property Value

double

ChosenLogProb

Gets or sets the chosen response log probability.

public double ChosenLogProb { get; set; }

Property Value

double

CustomMetrics

Gets or sets additional custom metrics.

public Dictionary<string, double> CustomMetrics { get; set; }

Property Value

Dictionary<string, double>

DistillationLoss

Gets or sets the distillation loss.

public double DistillationLoss { get; set; }

Property Value

double

FalseRefusalRate

Gets or sets the false refusal rate.

public double FalseRefusalRate { get; set; }

Property Value

double

Remarks

Measures how often the model incorrectly refuses benign requests. Lower is better - too high indicates over-refusal.

GroupAdvantage

Gets or sets the average group advantage.

public double GroupAdvantage { get; set; }

Property Value

double

GroupRewardVariance

Gets or sets the within-group reward variance.

public double GroupRewardVariance { get; set; }

Property Value

double

HarmlessnessScore

Gets or sets the harmlessness score.

public double HarmlessnessScore { get; set; }

Property Value

double

Remarks

Measures how often the model avoids harmful outputs.

Values: 0.0 to 1.0 (higher is better)

HelpfulnessScore

Gets or sets the helpfulness score.

public double HelpfulnessScore { get; set; }

Property Value

double

HonestyScore

Gets or sets the honesty score.

public double HonestyScore { get; set; }

Property Value

double

KLDivergence

Gets or sets the KL divergence from the reference model.

public double KLDivergence { get; set; }

Property Value

double

Remarks

Measures how much the fine-tuned model has diverged from the reference. High KL divergence may indicate over-optimization.

LogProbMargin

Gets or sets the margin between chosen and rejected log probabilities.

public double LogProbMargin { get; set; }

Property Value

double

LossHistory

Gets or sets the loss history over training.

public List<double> LossHistory { get; set; }

Property Value

List<double>

MethodName

Gets or sets the fine-tuning method used.

public string MethodName { get; set; }

Property Value

string

PeakMemoryGB

Gets or sets the peak memory usage in GB.

public double PeakMemoryGB { get; set; }

Property Value

double

Perplexity

Gets or sets the perplexity on validation data.

public double Perplexity { get; set; }

Property Value

double

PolicyEntropy

Gets or sets the entropy of the policy.

public double PolicyEntropy { get; set; }

Property Value

double

Remarks

Higher entropy indicates more exploration/diversity in outputs.

PolicyLoss

Gets or sets the policy loss.

public double PolicyLoss { get; set; }

Property Value

double

PreferenceAccuracy

Gets or sets the preference accuracy on held-out data.

public double PreferenceAccuracy { get; set; }

Property Value

double

Remarks

How accurately the model predicts which response is preferred.

RefusalRate

Gets or sets the refusal rate for harmful prompts.

public double RefusalRate { get; set; }

Property Value

double

Remarks

Measures how often the model appropriately refuses harmful requests.

RejectedLogProb

Gets or sets the rejected response log probability.

public double RejectedLogProb { get; set; }

Property Value

double

RewardStd

Gets or sets the reward standard deviation.

public double RewardStd { get; set; }

Property Value

double

RougeLScore

Gets or sets the ROUGE-L score.

public double RougeLScore { get; set; }

Property Value

double

TeacherAgreementRate

Gets or sets the agreement rate with teacher model.

public double TeacherAgreementRate { get; set; }

Property Value

double

Remarks

Measures how often the student agrees with the teacher's predictions.

ThroughputSamplesPerSecond

Gets or sets the throughput in samples per second.

public double ThroughputSamplesPerSecond { get; set; }

Property Value

double

TrainableParameters

Gets or sets the number of trainable parameters.

public long TrainableParameters { get; set; }

Property Value

long

TrainingEndTime

Gets or sets when the training completed.

public DateTime TrainingEndTime { get; set; }

Property Value

DateTime

TrainingLoss

Gets or sets the final training loss.

public double TrainingLoss { get; set; }

Property Value

double

TrainingStartTime

Gets or sets when the training started.

public DateTime TrainingStartTime { get; set; }

Property Value

DateTime

TrainingSteps

Gets or sets the number of training steps completed.

public int TrainingSteps { get; set; }

Property Value

int

TrainingTimeSeconds

Gets or sets the total training time in seconds.

public double TrainingTimeSeconds { get; set; }

Property Value

double

ValidationLoss

Gets or sets the validation loss.

public double ValidationLoss { get; set; }

Property Value

double

ValueLoss

Gets or sets the value function loss (for actor-critic methods).

public double ValueLoss { get; set; }

Property Value

double

WinRate

Gets or sets the win rate against the reference model.

public double WinRate { get; set; }

Property Value

double

Remarks

Measures how often the fine-tuned model is preferred over the reference. Used by DPO, SimPO, and other preference methods.

Values: 0.0 to 1.0 (higher is better)

Methods

GetSummary()

Gets a summary of the metrics suitable for logging.

public string GetSummary()

Returns

string

A formatted string with key metrics.