TL;DR
This paper reveals that a seemingly minor detail in how large language models (LLMs) process prompts for multiple-choice question answering – specifically, how the space before the answer letter is tokenized – can significantly impact performance and even alter model rankings. The authors demonstrate that tokenizing the space with the answer letter consistently improves accuracy and calibration, urging for standardized evaluation protocols.
What Problem Does This Paper Address?
Evaluating LLMs, especially on tasks like multiple-choice question answering (MCQA), relies heavily on how prompts are structured and how models interpret them. A common practice is to end a prompt with "Answer:" and then analyze the probabilities of the subsequent tokens representing the answer choices (e.g., A, B, C, D). However, there's no consensus on whether the space immediately following "Answer:" should be treated as a separate token or grouped with the answer letter. This ambiguity, often overlooked as trivial, can lead to inconsistent and potentially misleading comparisons of LLM capabilities across different studies and evaluation frameworks. The paper aims to quantify the impact of this tokenization choice and provide a recommendation for more reliable LLM evaluations.
Methods
The core of the paper's methodology involves comparing two distinct tokenization strategies for the end of MCQA prompts:
- Letter Token Without Space: The space after "Answer:" is tokenized as a separate token, and the model's prediction is based on the probability of the answer letter token itself (e.g., "Answer:" -> " ", "A").
- Space-Letter Token: The space and the answer letter are tokenized together as a single token (e.g., "Answer:" -> " A").
The authors evaluate LLMs on MCQA tasks, measuring both performance (accuracy) and calibration (how well the model's confidence estimates reflect its actual accuracy, using Expected Calibration Error - ECE). They employ statistical tests (McNemar's test for accuracy, paired bootstrap resampling for ECE) to determine the significance of observed differences. They also explore the impact of this tokenization choice across various prompt variations (e.g., different option formatting, few-shot learning, chain-of-thought prompting) and languages to assess robustness.
Datasets
The primary dataset used for evaluation is MMLU (Massive Multitask Language Understanding), a widely used benchmark covering 57 diverse fields. Additionally, the study evaluates on five other commonly used MCQA datasets:
- ARC Challenge
- ARC Easy
- HellaSwag
- OpenbookQA
- TruthfulQA
No new benchmarks were introduced; the focus was on analyzing existing ones.
Key Insights and Ideas
- The "Trivial" Space Matters: The most striking insight is that the tokenization of the space before the answer letter is far from trivial. It can lead to accuracy differences of up to 11% and significantly alter model rankings on leaderboards.
- Space-Letter Tokenization Wins: The "Space-Letter" tokenization strategy (grouping the space with the answer letter) consistently and significantly improves both accuracy and calibration across a wide range of LLMs and datasets.
- Impact on Leaderboards: This subtle change can flip the top-performing model in evaluations, raising serious concerns about the comparability and reliability of existing LLM leaderboards.
- Calibration Boost: Beyond accuracy, the "Space-Letter" tokenization also improves model calibration, making their confidence scores more trustworthy. This is crucial for understanding how reliable a model's prediction truly is.
- Robustness Across Variations: The positive effect of "Space-Letter" tokenization is observed across different prompt formats (e.g., parentheses, numeric labels), few-shot settings, and even in languages not natively supported by the models, highlighting its general applicability.
Key Results
- Accuracy Gains: The "Space-Letter" tokenization strategy led to accuracy improvements of up to 11% on some datasets (e.g., HellaSwag for Qwen 2.5 72B).
- Model Ranking Shifts: For instance, Llama 3.1 70B Instruct topped the leaderboard with one tokenization, while Qwen 2.5 72B took the lead with the other.
- Calibration Improvements: ECE reductions were observed for the majority of models, with some showing up to 4 times more reliable answers (e.g., Gemma 3 4B).
- Statistical Significance: The observed improvements in accuracy and calibration were statistically significant for most models and datasets (p < 0.05).
Limitations and Future Work
- Model Scope: The study primarily focused on open-weight models due to the need for access to next-token logits. API-based models were not included. The experiments used LLMs up to 72B parameters, and testing with larger-scale models is suggested for future work.
- Standardization Need: The authors strongly advocate for standardized, transparent evaluation protocols to ensure fair and comparable LLM comparisons.
- Broader Impact: While the study focused on MCQA, future work could explore if similar tokenization effects impact other NLP tasks.
Summary updated