Tokenizer Vocabulary
Definition
The set of unique token subwords recognized by a model tokenizer.
Example Case
Llama 3 using a 128k token vocabulary.
The set of unique token subwords recognized by a model tokenizer.