Class BenchmarkingOptions

Namespace

AiDotNet.Configuration

Assembly

AiDotNet.dll

Configuration options for running benchmarks through the AiDotNet facade.

public sealed class BenchmarkingOptions

Inheritance

object

BenchmarkingOptions

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This options class is designed for use with AiModelBuilder.ConfigureBenchmarking(...) and AiModelResult.EvaluateBenchmarksAsync(...). Users specify which benchmark suites to run, and AiDotNet orchestrates the execution behind the scenes.

For Beginners: Benchmarking is how you measure model quality using standardized tests. You choose a suite (like GSM8K for math reasoning), and AiDotNet runs it and returns a report.

Properties

AttachReportToResult

Gets or sets whether the generated report should be attached to the model result.

public bool AttachReportToResult { get; set; }

Property Value

bool

Remarks

When true, the last produced report is stored on the AiModelResult instance and can be retrieved via AiModelResult.GetBenchmarkReport().

CiMode

Gets or sets whether benchmarking should run in CI-friendly mode.

public bool CiMode { get; set; }

Property Value

bool

Remarks

CI-friendly mode favors smaller sample sizes to keep runs fast and stable. Deterministic seeding is introduced in Sprint 2 (dataset suites + CI mode).

DetailLevel

Gets or sets how much detail should be included in reports.

public BenchmarkReportDetailLevel DetailLevel { get; set; }

Property Value

BenchmarkReportDetailLevel

FailurePolicy

Gets or sets how failures should be handled.

public BenchmarkFailurePolicy FailurePolicy { get; set; }

Property Value

BenchmarkFailurePolicy

Leaf

Gets or sets LEAF federated benchmark configuration (required when running LEAF).

public LeafFederatedBenchmarkOptions? Leaf { get; set; }

Property Value

LeafFederatedBenchmarkOptions

SampleSize

Gets or sets an optional sample size for suites that support sampling.

public int? SampleSize { get; set; }

Property Value

int?

Remarks

If null, suites use their default behavior (often "all problems" or suite-defined sampling).

Seed

Gets or sets an optional deterministic seed used for CI-friendly sampling.

public int? Seed { get; set; }

Property Value

int?

Remarks

When specified, suites that support subsampling use this value to ensure repeatable selection.

Suites

Gets or sets the benchmark suites to run.

public BenchmarkSuite[] Suites { get; set; }

Property Value

BenchmarkSuite[]

Remarks

If empty, no benchmarks will be run.

Tabular

Gets or sets federated tabular benchmark configuration (synthetic non-IID suite).

public FederatedTabularBenchmarkOptions? Tabular { get; set; }

Property Value

FederatedTabularBenchmarkOptions

Text

Gets or sets federated text benchmark configuration (Sent140/Shakespeare suites).

public FederatedTextBenchmarkOptions? Text { get; set; }

Property Value

FederatedTextBenchmarkOptions

Vision

Gets or sets federated vision benchmark configuration (FEMNIST/CIFAR suites).

public FederatedVisionBenchmarkOptions? Vision { get; set; }

Property Value

FederatedVisionBenchmarkOptions