Class ProgramSynthesisTokenizerFactory

Namespace

AiDotNet.ProgramSynthesis.Tokenization

Assembly

AiDotNet.dll

Creates safe, code-aware default tokenizers for ProgramLanguage values.

public static class ProgramSynthesisTokenizerFactory

Inheritance

object

ProgramSynthesisTokenizerFactory

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

CreateDefault(ProgramLanguage, bool)

public static ITokenizer CreateDefault(ProgramLanguage language, bool splitIdentifiers = true)

Parameters

language ProgramLanguage

splitIdentifiers bool

Returns

ITokenizer