Class ParallelBatchLoaderExtensions

Namespace

AiDotNet.Data.Loaders

Assembly

AiDotNet.dll

Provides extension methods for parallel batch loading.

public static class ParallelBatchLoaderExtensions

Inheritance

object

ParallelBatchLoaderExtensions

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

WithParallelLoading<TBatch>(IBatchIterable<TBatch>, Func<int[], TBatch>, int, int?, int?)

Wraps a batch iterable with parallel loading support.

public static ParallelBatchLoader<TBatch> WithParallelLoading<TBatch>(this IBatchIterable<TBatch> source, Func<int[], TBatch> batchFactory, int batchSize, int? numWorkers = null, int? prefetchCount = null)

Parameters

source IBatchIterable<TBatch>

The source batch iterable.

batchFactory Func<int[], TBatch>

Function to create a batch from an array of sample indices. The factory receives all indices for a single batch and should aggregate them.

batchSize int

The batch size (number of samples per batch).

numWorkers int?

Number of parallel workers. Default is processor count.

prefetchCount int?

Number of batches to prefetch. Default is 2 * numWorkers.

Returns

ParallelBatchLoader<TBatch>

A parallel batch loader configured with the specified parameters.

Type Parameters

TBatch

The batch type.

Examples

var parallelLoader = dataLoader.WithParallelLoading(
    batchFactory: indices => {
        var xBatch = new Matrix<float>(indices.Length, numFeatures);
        var yBatch = new Vector<float>(indices.Length);
        for (int i = 0; i < indices.Length; i++) {
            int idx = indices[i];
            // Copy sample data at idx to position i in batch
            for (int j = 0; j < numFeatures; j++)
                xBatch[i, j] = fullDataset.X[idx, j];
            yBatch[i] = fullDataset.Y[idx];
        }
        return (xBatch, yBatch);
    },
    batchSize: 32,
    numWorkers: 4
);