Token
Definition
The basic unit of text processed by an LLM, typically representing a word, sub-word, or character sequence.
Example Case
The word "AI" might represent 1 token; "unbelievable" might represent 3 tokens.
The basic unit of text processed by an LLM, typically representing a word, sub-word, or character sequence.