Namespace AiDotNet.ProgramSynthesis.Tokenization
Classes
- CodeTokenizationPipeline
Default implementation of ICodeTokenizationPipeline.
- CodeTokenizationPipelineOptions
Options for CodeTokenizationPipeline that control optional structural extraction.
- ProgramSynthesisTokenizerFactory
Creates safe, code-aware default tokenizers for ProgramLanguage values.
Interfaces
- ICodeTokenizationPipeline
Defines a code tokenization pipeline that builds on the core tokenizer stack and adds code-oriented metadata.