Tokenizer
Definition
A mathematical parser that converts raw text strings into arrays of token integer IDs matching a model’s vocabulary.
Example Case
Tiktoken for OpenAI models; LlamaTokenizer for Meta models.
A mathematical parser that converts raw text strings into arrays of token integer IDs matching a model’s vocabulary.