kmplitert
Toggle table of contents
common
Target filter
common
Switch theme
Search in API
Skip to content
kmplitert
tool
/
io.github.kmplitert.tool
/
LiteRTExt
/
Nlp
/
Tokenizer
/
tokenize
tokenize
abstract
fun
tokenize
(
text
:
String
)
:
List
<
Int
>
Converts a string into a list of token IDs.
Return
A list of integer token IDs.
Parameters
text
The input text.