Interface ICodeTokenizationPipeline

Namespace

AiDotNet.ProgramSynthesis.Tokenization

Assembly

AiDotNet.dll

Defines a code tokenization pipeline that builds on the core tokenizer stack and adds code-oriented metadata.

public interface ICodeTokenizationPipeline

Methods

Tokenize(string, ProgramLanguage, ITokenizer, EncodingOptions?, string?)

CodeTokenizationResult Tokenize(string code, ProgramLanguage language, ITokenizer tokenizer, EncodingOptions? options = null, string? filePath = null)

Parameters

code string

language ProgramLanguage

tokenizer ITokenizer

options EncodingOptions

filePath string

Returns

CodeTokenizationResult