tool/io.github.kmplitert.tool/LiteRTExt/Nlp/Tokenizer

Tokenizer

interface Tokenizer

Interface for KMP-compatible tokenizers.

Tokenizers convert text into a sequence of tokens (integers) processed by models like BERT.

Properties

abstract val vocabSize: Int

Returns the vocabulary size of the tokenizer.

Functions

abstract fun detokenize(tokens: List<Int>): String

Converts a list of token IDs back into a string.

abstract fun tokenize(text: String): List<Int>

Converts a string into a list of token IDs.