Class InferenceContext<T>

Namespace

AiDotNet.Memory

Assembly

AiDotNet.dll

Provides a scoped context for inference operations with automatic tensor pooling and lifecycle management. All tensors rented through this context are tracked and automatically returned to the pool when disposed.

public class InferenceContext<T> : IDisposable

Type Parameters

T

The numeric type for tensor elements (e.g., float, double).

Inheritance

object

InferenceContext<T>

Implements

IDisposable

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

InferenceContext simplifies memory management during neural network inference by: - Tracking all rented tensors automatically - Returning all tensors to the pool on disposal (even if Release wasn't called) - Providing convenient rent methods for common tensor shapes

Basic usage example:

using var context = new InferenceContext<float>();
var input = context.Rent2D(32, 784);    // Rent a batch of inputs
var hidden = context.Rent2D(32, 256);   // Rent hidden layer buffer
var output = context.Rent2D(32, 10);    // Rent output buffer
// Perform inference operations...
// All tensors automatically returned when context is disposed

For ambient context support (avoiding parameter threading), use InferenceScope<T>:

using var context = new InferenceContext<float>();
using var scope = InferenceScope<float>.Begin(context);
// Now any code can access the context via InferenceScope<float>.Current
var tensor = InferenceScope<float>.RentOrCreate(new[] { 32, 784 });

Constructors

InferenceContext(TensorPool<T>?, int)

Initializes a new instance of the InferenceContext<T> class.

public InferenceContext(TensorPool<T>? pool = null, int maxPoolSizeMB = 256)

Parameters

pool TensorPool<T>

An existing tensor pool to use. If null, a new pool is created and owned by this context.

maxPoolSizeMB int

The maximum pool size in MB if creating a new pool. Ignored if pool is provided.

Remarks

When a pool is provided, the context does not own it and will not dispose it. When no pool is provided, the context creates and owns its own pool.

Properties

IsPoolingEnabled

Gets or sets whether pooling is enabled. When disabled, tensors are allocated directly without pool reuse, which can be useful for debugging.

public bool IsPoolingEnabled { get; set; }

Property Value

bool

Pool

Gets the underlying tensor pool used by this context.

public TensorPool<T> Pool { get; }

Property Value

TensorPool<T>

RentedTensorCount

Gets the number of tensors currently rented from this context.

public int RentedTensorCount { get; }

Property Value

int

Methods

Dispose()

Disposes the context and returns all rented tensors to the pool.

public void Dispose()

Dispose(bool)

Disposes the context resources.

protected virtual void Dispose(bool disposing)

Parameters

disposing bool

True if called from Dispose(), false if called from finalizer.

Release(Tensor<T>)

Releases a tensor back to the pool before the context is disposed. Use this for early release of tensors that are no longer needed.

public void Release(Tensor<T> tensor)

Parameters

tensor Tensor<T>

The tensor to release.

Remarks

Releasing tensors early can reduce peak memory usage during long inference operations. Tensors not explicitly released will be automatically returned when the context is disposed.

Exceptions

ObjectDisposedException

Thrown if the context has been disposed.

Rent(int[])

Rents a tensor with the specified shape from the pool. The tensor is tracked and will be automatically returned when this context is disposed.

public Tensor<T> Rent(int[] shape)

Parameters

shape int[]

The shape of the tensor to rent.

Returns

Tensor<T>

A tensor with the requested shape.

Exceptions

ObjectDisposedException

Thrown if the context has been disposed.

Rent1D(int)

Rents a 1D tensor (vector) with the specified length.

public Tensor<T> Rent1D(int length)

Parameters

length int

The number of elements in the tensor.

Returns

Tensor<T>

A 1D tensor with shape [length].

Rent2D(int, int)

Rents a 2D tensor (matrix) with the specified dimensions.

public Tensor<T> Rent2D(int rows, int cols)

Parameters

rows int

The number of rows.

cols int

The number of columns.

Returns

Tensor<T>

A 2D tensor with shape [rows, cols].

Rent3D(int, int, int)

Rents a 3D tensor with the specified dimensions.

public Tensor<T> Rent3D(int dim0, int dim1, int dim2)

Parameters

dim0 int

The first dimension (e.g., batch size).

dim1 int

The second dimension (e.g., sequence length).

dim2 int

The third dimension (e.g., feature size).

Returns

Tensor<T>

A 3D tensor with shape [dim0, dim1, dim2].

Rent4D(int, int, int, int)

Rents a 4D tensor with the specified dimensions (typically for image data).

public Tensor<T> Rent4D(int batch, int channels, int height, int width)

Parameters

batch int

The batch size.

channels int

The number of channels (e.g., RGB = 3).

height int

The image height.

width int

The image width.

Returns

Tensor<T>

A 4D tensor with shape [batch, channels, height, width].

RentLike(Tensor<T>)

Rents a tensor with the same shape as the template tensor.

public Tensor<T> RentLike(Tensor<T> template)

Parameters

template Tensor<T>

The tensor whose shape should be matched.

Returns

Tensor<T>

A tensor with the same shape as the template.

Exceptions

ArgumentNullException

Thrown if template is null.