Namespace AiDotNet.Tokenization.HuggingFace

Classes

AutoTokenizer

AutoTokenizer provides HuggingFace-style automatic tokenizer loading. This class automatically detects and loads the appropriate tokenizer type based on the model configuration.

HuggingFaceTokenizerLoader

Loads HuggingFace pretrained tokenizers.

TokenizerConfig

Configuration for HuggingFace tokenizers.