Table of Contents

Class CachingCompressor

Namespace
AiDotNet.PromptEngineering.Compression
Assembly
AiDotNet.dll

Wrapper compressor that caches compression results for frequently used prompts.

public class CachingCompressor : PromptCompressorBase, IPromptCompressor
Inheritance
CachingCompressor
Implements
Inherited Members

Remarks

This compressor wraps another compressor and caches the results. When the same prompt is compressed multiple times, the cached result is returned instead of re-computing the compression. This is particularly useful for LLM-based compressors where each compression call is expensive.

For Beginners: Remembers compressions so the same prompt doesn't need to be compressed twice.

Example:

var llmCompressor = new LLMSummarizationCompressor(myLlmFunc);
var cachingCompressor = new CachingCompressor(llmCompressor, maxCacheSize: 1000);

// First call - compresses using LLM (slow)
var result1 = await cachingCompressor.CompressAsync(prompt);

// Second call with same prompt - returns cached result (fast)
var result2 = await cachingCompressor.CompressAsync(prompt);

// result1 == result2, but second call was instant

When to use:

  • Wrapping expensive compressors (LLM-based)
  • When the same prompts are used repeatedly
  • In production systems to reduce API calls

Constructors

CachingCompressor(IPromptCompressor, int, TimeSpan?, Func<string, int>?)

Initializes a new instance of the CachingCompressor class.

public CachingCompressor(IPromptCompressor innerCompressor, int maxCacheSize = 1000, TimeSpan? cacheExpiry = null, Func<string, int>? tokenCounter = null)

Parameters

innerCompressor IPromptCompressor

The compressor to wrap with caching.

maxCacheSize int

Maximum number of entries in the cache.

cacheExpiry TimeSpan?

How long cached entries remain valid.

tokenCounter Func<string, int>

Optional custom token counter function.

Properties

CacheCount

Gets the number of items currently in the cache.

public int CacheCount { get; }

Property Value

int

CacheHitRatio

Gets the cache hit ratio (hits / total requests).

public double CacheHitRatio { get; }

Property Value

double

Methods

ClearCache()

Clears all entries from the cache.

public void ClearCache()

CompressAsync(string, CompressionOptions?, CancellationToken)

Compresses the prompt asynchronously, using cache if available.

public override Task<string> CompressAsync(string prompt, CompressionOptions? options = null, CancellationToken cancellationToken = default)

Parameters

prompt string
options CompressionOptions
cancellationToken CancellationToken

Returns

Task<string>

CompressCore(string, CompressionOptions)

Compresses the prompt, using cache if available.

protected override string CompressCore(string prompt, CompressionOptions options)

Parameters

prompt string
options CompressionOptions

Returns

string

GetCacheStats()

Gets statistics about the cache.

public Dictionary<string, object> GetCacheStats()

Returns

Dictionary<string, object>

A dictionary with cache statistics.

PurgeExpiredEntries()

Removes expired entries from the cache.

public int PurgeExpiredEntries()

Returns

int

The number of entries removed.

Edit this page