Class TextProcessingHelper

Namespace

AiDotNet.Helpers

Assembly

AiDotNet.dll

Provides text processing utilities for splitting and tokenizing text.

public static class TextProcessingHelper

Inheritance

object

TextProcessingHelper

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

SplitIntoSentences(string)

Splits text into sentences based on common sentence-ending punctuation. Handles periods, exclamation marks, and question marks followed by spaces or newlines.

public static List<string> SplitIntoSentences(string text)

Parameters

text string

The text to split into sentences.

Returns

List<string>

A list of sentences extracted from the text.

Tokenize(string)

Tokenizes text by splitting on whitespace and common punctuation marks. Converts text to lowercase and removes empty tokens.

public static List<string> Tokenize(string text)

Parameters

text string

The text to tokenize.

Returns

List<string>

A list of tokens extracted from the text.