Table of Contents

Class ConfidenceBasedAdaptiveStrategy<T>

Namespace
AiDotNet.KnowledgeDistillation.Strategies
Assembly
AiDotNet.dll

Adaptive distillation strategy that adjusts temperature based on student confidence.

public class ConfidenceBasedAdaptiveStrategy<T> : AdaptiveDistillationStrategyBase<T>, IDistillationStrategy<T>, IAdaptiveDistillationStrategy<T>

Type Parameters

T

The numeric type for calculations (e.g., double, float).

Inheritance
ConfidenceBasedAdaptiveStrategy<T>
Implements
Inherited Members

Remarks

For Beginners: This strategy adapts temperature based on how confident the student is in its predictions. When the student is confident (high max probability), we use lower temperature (harder distillation). When uncertain (low max probability), we use higher temperature (easier distillation with softer targets).

Intuition: - **High Confidence** → Student understands this sample → Lower temp (sharpen targets) - **Low Confidence** → Student struggles with this sample → Higher temp (soften targets)

Example: Student predicts [0.95, 0.03, 0.02] → High confidence (0.95) → Low temperature Student predicts [0.40, 0.35, 0.25] → Low confidence (0.40) → High temperature

Best For: - General-purpose adaptive distillation - When you want automatic difficulty adjustment - Datasets with varying sample complexity

Temperature Mapping: Confidence = max(probabilities) Difficulty = 1 - Confidence Temperature = MinTemp + Difficulty * (MaxTemp - MinTemp)

Constructors

ConfidenceBasedAdaptiveStrategy(double, double, double, double, double)

Initializes a new instance of the ConfidenceBasedAdaptiveStrategy class.

public ConfidenceBasedAdaptiveStrategy(double baseTemperature = 3, double alpha = 0.3, double minTemperature = 1, double maxTemperature = 5, double adaptationRate = 0.1)

Parameters

baseTemperature double

Base temperature for distillation (default: 3.0).

alpha double

Balance between hard and soft loss (default: 0.3).

minTemperature double

Minimum temperature (for high confidence samples, default: 1.0).

maxTemperature double

Maximum temperature (for low confidence samples, default: 5.0).

adaptationRate double

EMA rate for performance tracking (default: 0.1).

Remarks

For Beginners: The default parameters work well for most cases. Adjust minTemperature and maxTemperature to control the adaptation range.

Example: 

// Conservative adaptation (narrow temperature range)
var strategy1 = new ConfidenceBasedAdaptiveStrategy<double>(
    minTemperature: 2.0,
    maxTemperature: 4.0
);

// Aggressive adaptation (wide temperature range)
var strategy2 = new ConfidenceBasedAdaptiveStrategy<double>(
minTemperature: 1.0,
maxTemperature: 8.0
);

Methods

ComputeAdaptiveTemperature(Vector<T>, Vector<T>)

Computes adaptive temperature based on student confidence.

public override double ComputeAdaptiveTemperature(Vector<T> studentOutput, Vector<T> teacherOutput)

Parameters

studentOutput Vector<T>

Student's output logits.

teacherOutput Vector<T>

Teacher's output logits (not used in confidence-based).

Returns

double

Adapted temperature based on student confidence.

Remarks

Algorithm: 1. Convert logits to probabilities (softmax) 2. Find maximum probability (confidence) 3. Compute difficulty = 1 - confidence 4. Map to temperature range: temp = min + difficulty * (max - min)

This creates an inverse relationship: - High confidence (0.9) → Low difficulty (0.1) → Low temperature (~1.4 with defaults) - Low confidence (0.4) → High difficulty (0.6) → High temperature (~3.4 with defaults)

Edit this page