Class StopWordRemovalCompressor

Namespace

AiDotNet.PromptEngineering.Compression

Assembly

AiDotNet.dll

Compressor that removes common stop words to reduce prompt length.

public class StopWordRemovalCompressor : PromptCompressorBase, IPromptCompressor

Inheritance

object

PromptCompressorBase

StopWordRemovalCompressor

Implements

IPromptCompressor

Inherited Members

PromptCompressorBase.Name

PromptCompressorBase.Compress(string, CompressionOptions)

PromptCompressorBase.CompressWithMetrics(string, CompressionOptions)

PromptCompressorBase.CompressAsync(string, CompressionOptions, CancellationToken)

PromptCompressorBase.CountTokens(string)

PromptCompressorBase.ValidatePrompt(string)

PromptCompressorBase.ExtractVariables(string)

PromptCompressorBase.ReplaceVariablesWithPlaceholders(string, Dictionary<string, string>)

PromptCompressorBase.RestoreVariables(string, Dictionary<string, string>)

PromptCompressorBase.ExtractCodeBlocks(string)

PromptCompressorBase.ReplaceCodeBlocksWithPlaceholders(string, Dictionary<string, string>)

PromptCompressorBase.RestoreCodeBlocks(string, Dictionary<string, string>)

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This compressor removes frequently occurring words that often don't add significant meaning to the prompt (articles, prepositions, auxiliary verbs, etc.). It's more aggressive than redundancy removal and may slightly reduce readability.

For Beginners: Removes common words like "the", "a", "is" to make prompts shorter.

Example:

var compressor = new StopWordRemovalCompressor();

string original = "Please analyze the document and provide a summary of the main points";
string compressed = compressor.Compress(original);
// Result: "analyze document provide summary main points"

When to use:

• When maximum compression is needed
• When the AI can infer meaning from keywords
• Not recommended for conversational prompts

Constructors

StopWordRemovalCompressor(AggressivenessLevel, Func<string, int>?)

Initializes a new instance of the StopWordRemovalCompressor class.

public StopWordRemovalCompressor(StopWordRemovalCompressor.AggressivenessLevel level = AggressivenessLevel.Medium, Func<string, int>? tokenCounter = null)

Parameters

level StopWordRemovalCompressor.AggressivenessLevel

The aggressiveness level for stop word removal.

tokenCounter Func<string, int>

Optional custom token counter function.

Methods

CompressCore(string, CompressionOptions)

Compresses the prompt by removing stop words.

protected override string CompressCore(string prompt, CompressionOptions options)

Parameters

prompt string

options CompressionOptions

Returns

string