Interface IAdaptiveDistillationStrategy<T>

Namespace

AiDotNet.KnowledgeDistillation.Strategies

Assembly

AiDotNet.dll

Interface for adaptive distillation strategies that adjust temperature based on student performance.

public interface IAdaptiveDistillationStrategy<T>

Type Parameters

T

The numeric type for calculations (e.g., double, float).

Remarks

For Beginners: Adaptive strategies dynamically adjust the temperature parameter during training based on how well the student is learning. This allows for more flexible knowledge transfer compared to fixed-temperature distillation.

Key Concepts: - **Performance Tracking**: Monitor student learning progress - **Temperature Adaptation**: Adjust temperature based on sample difficulty or student confidence - **Per-Sample Adjustment**: Different temperatures for different training samples

When to Use: - Training data has varying difficulty levels - Student performance is uneven across samples - You want automatic temperature tuning instead of manual selection

Properties

AdaptationRate

Gets the adaptation rate for performance tracking.

double AdaptationRate { get; }

Property Value

double

Remarks

Controls how quickly the strategy adapts to new performance metrics. Lower values = slower adaptation, Higher values = faster adaptation.

MaxTemperature

Gets the maximum temperature value used for adaptation.

double MaxTemperature { get; }

Property Value

double

Remarks

Higher temperatures produce softer probability distributions, making the distillation easier for the student.

MinTemperature

Gets the minimum temperature value used for adaptation.

double MinTemperature { get; }

Property Value

double

Remarks

Lower temperatures produce sharper probability distributions, making the distillation more challenging for the student.

Methods

ComputeAdaptiveTemperature(Vector<T>, Vector<T>)

Computes the adaptive temperature for a specific sample.

double ComputeAdaptiveTemperature(Vector<T> studentOutput, Vector<T> teacherOutput)

Parameters

studentOutput Vector<T>

Student's current output (logits).

teacherOutput Vector<T>

Teacher's output (logits).

Returns

double

Adapted temperature value for this sample.

Remarks

For Implementers: This method should analyze the student's output and return an appropriate temperature within [MinTemperature, MaxTemperature].

Common approaches: - High confidence → lower temperature (student is doing well) - Low confidence → higher temperature (student needs help) - High entropy → lower temperature (focus learning) - Low entropy → higher temperature (explore more)

GetPerformance(int)

Gets the current performance metric for a specific sample.

double GetPerformance(int sampleIndex)

Parameters

sampleIndex int

Index of the sample.

Returns

double

Performance metric (interpretation depends on strategy).

Remarks

Returns the tracked performance for a sample, or 0.5 (neutral) if not yet tracked.

UpdatePerformance(int, Vector<T>, Vector<T>?)

Updates the performance metric for a specific training sample.

void UpdatePerformance(int sampleIndex, Vector<T> studentOutput, Vector<T>? trueLabel = null)

Parameters

sampleIndex int

Index of the training sample.

studentOutput Vector<T>

Student's output (logits) for this sample.

trueLabel Vector<T>

Optional true label for accuracy-based tracking.

Remarks

For Beginners: Call this method after each training step to update the strategy's understanding of how well the student is performing on each sample.

The strategy uses this information to adjust temperature in future iterations.