Class CodeTokenizationResult

Namespace

AiDotNet.ProgramSynthesis.Models

Assembly

AiDotNet.dll

Represents the result of code-aware tokenization, including token IDs and token-to-source spans.

public sealed class CodeTokenizationResult

Inheritance

object

CodeTokenizationResult

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This type is designed to reuse the existing tokenization stack (ITokenizer) while providing code-specific structural metadata (spans) that is useful for downstream tasks (search, review, diagnostics, etc.).

Properties

AstEdges

Optional relationships between AST nodes (when enabled).

public List<CodeAstEdge> AstEdges { get; set; }

Property Value

List<CodeAstEdge>

AstNodes

Optional AST nodes extracted during tokenization (when enabled and supported for the language).

public List<CodeAstNode> AstNodes { get; set; }

Property Value

List<CodeAstNode>

FilePath

public string? FilePath { get; set; }

Property Value

string

Language

public ProgramLanguage Language { get; set; }

Property Value

ProgramLanguage

TokenSpans

Best-effort mapping from each token to a span in the original source code.

public List<CodeSpan> TokenSpans { get; set; }

Property Value

List<CodeSpan>

Remarks

When the underlying tokenizer does not provide offsets, this may be empty. When offsets are provided, this is aligned with Tokens.

Tokenization

public TokenizationResult Tokenization { get; set; }

Property Value

TokenizationResult