Namespace AiDotNet.Tokenization.Configuration
Classes
- PretrainedTokenizerModelExtensions
Extension methods for PretrainedTokenizerModel enum.
- TokenizationConfig
Configuration options for tokenization in the prediction pipeline.
Enums
- PretrainedTokenizerModel
Specifies pretrained tokenizer models available from HuggingFace Hub.