Throughput
Definition
The speed of generation measured in tokens per second processed by the serving server.
Example Case
vLLM outputting 250 tokens/sec across 10 concurrent users.
The speed of generation measured in tokens per second processed by the serving server.