Namespace AiDotNet.Tokenization.HuggingFace
Classes
- AutoTokenizer
AutoTokenizer provides HuggingFace-style automatic tokenizer loading. This class automatically detects and loads the appropriate tokenizer type based on the model configuration.
- HuggingFaceTokenizerLoader
Loads HuggingFace pretrained tokenizers.
- TokenizerConfig
Configuration for HuggingFace tokenizers.