TL;DR
When choosing an LLM, we often look at "tokens per second" as the gold standard for speed. This paper reveals a major catch: raw token throughput poorly predicts actual task completion time because different models use different tokenizers and vary wildly in their verbosity. To get a true sense of performance, we need to measure the time it takes to complete specific tasks rather than how fast a model spits out individual tokens.
What Problem Does This Paper Address?
In the rapidly growing ecosystem of open-weights Large Language Models (LLMs), developers and researchers heavily rely on token-based metrics to compare model speeds. However, the authors argue that this approach has two critical flaws:
- Tokenizer Disparity: Different models use different tokenizers, meaning the exact same text maps to a different number of tokens depending on the model family.
- Model Verbosity: Models exhibit different levels of wordiness. One model might answer a question in a single sentence, while another writes a full paragraph.
Because end-users ultimately care about how long it takes to get a complete answer, evaluating speed purely on a per-token basis creates an unfair and misleading comparison. This paper addresses the need for a task-oriented approach to benchmarking LLM speed.
Methods
The authors benchmarked five popular open-weights LLMs of similar sizes: LLaMa-2 (7B), LLaMa-3 (8B), Gemma (7B), Mistral (7B), and Yi (6B). To ensure a fair hardware baseline, all models were run in a 16-bit floating-point format on a single high-end NVIDIA A100 GPU (40GB).
To isolate how different tasks impact speed, the authors designed three distinct evaluation scenarios using the exact same set of underlying prompts:
- Objective Selection: The model is prompted to output only the correct option (e.g., a, b, c, d) for a multiple-choice question.
- Paraphrasing: The model rewrites the question, constraining the output length to closely match the input length.
- Open Answering: The model answers freely and provides an explanation, allowing for unconstrained text generation.
They also evaluated the models across different batch sizes (from 1 to 8) to observe how parallel processing affects token generation efficiency.
Datasets
The evaluation utilized 660 miscellaneous questions drawn from a standard multiple-choice LLM benchmark (MMLU). No new datasets were introduced; instead, the innovation lies in repurposing these questions into three distinct task types to analyze generation behavior and execution time.
Key Insights and Ideas
- The Verbosity Paradox: A model can be slow on a per-token basis but still finish the overall task faster if it is concise. For instance, a model that strictly follows instructions and avoids rambling will beat a faster token-generator that writes too much.
- Tokenizer Inconsistency: Because input token counts vary by tokenizer, comparing tokens/sec across different model families is fundamentally an apples-to-oranges comparison.
- Task-Dependent Rankings: A model's speed rank changes depending on what you ask it to do. LLaMa-2-7B is highly efficient at raw token generation speed, yet it falls behind in open-ended tasks because it tends to generate excessively long responses.
Key Results
- In Task 1 (Multiple Choice), LLaMa-3-8B and Yi-6B generated only 5,000–7,000 output tokens combined across all questions, compared to LLaMa-2-7B, which generated over 27,000 tokens. As a result, despite having a higher per-token latency, LLaMa-3 and Yi completed the task much faster.
- In Task 3 (Open Answer), Gemma-7B emerged as the fastest model to complete the entire task. Even though LLaMa-2-7B boasted a faster raw token generation rate, Gemma's concise responses meant it had significantly fewer total tokens to process, leading to a shorter overall execution time.
Limitations and Future Work
The study focused exclusively on models in the ~7B parameter range and evaluated them using 16-bit precision on a specific GPU architecture. The authors did not explore the impact of lower-bit quantization (like 4-bit or 8-bit formats), which are common in local commodity deployments. Additionally, generation length can be heavily influenced by hyperparameters like temperature, which wasn't deeply varied here.
Moving forward, the authors advocate for the creation of task-oriented speed benchmarks (focusing on translation, summarization, or essay writing) that measure total execution time rather than raw token throughput, providing a more realistic and comprehensive picture of LLM efficiency and energy consumption.
Summary updated