Abstract
Multiple-choice question answering (MCQA) is a cornerstone of LLM evaluation, yet models often show extreme sensitivity to label formatting and choice ordering. This study investigates the internal mechanics of this process using interpretability techniques like activation patching and logit lens. We find that "symbol binding"—the mapping of a conceptual answer to a specific label—is not a distributed computation but a localized two-stage process driven by a sparse set of attention heads in the model's middle layers. Our results explain why models struggle with non-standard labels and pinpoint the specific training milestones where these formatting skills emerge.
Introduction
Benchmarks like MMLU and HellaSwag are standard for measuring LLM performance. However, these models are notoriously finicky: changing labels from A/B/C/D to Q/Z/R/X or shuffling choices can significantly degrade performance. This raises a fundamental question: does the model truly "know" the answer, or is it failing at the mechanical task of formatting? This research localizes where and how a model decides that a specific symbol is the correct answer.
Methodology
The researchers analyzed models including Llama 3.1, Olmo, and Qwen 2.5 using three primary techniques:
- Activation Patching (Causal Tracing): Swapping internal states between "correct" and "modified" runs to identify layers responsible for the final prediction.
- Vocabulary Projection (Logit Lens): Projecting hidden states from every layer into the final vocabulary space to observe the model's internal "thought process."
- Logits vs. Probits Analysis: Distinguishing between general answer-choice boosting and specific winner selection.
The study utilized standard benchmarks (MMLU, HellaSwag) alongside a synthetic "Colors" task. In the Colors task, models must map a known fact (e.g., "Corn is yellow") to a label. This isolates the symbol-binding mechanism from complex reasoning or external knowledge gaps.
Key Insights: The Middle-Layer "Aha!" Moment
The research reveals that answer selection is not a gradual build-up. Instead, the "aha moment" occurs almost entirely within 1–3 middle layers (e.g., layer 24 in a 32-layer Olmo model).
Attention-Driven Sparsity
The heavy lifting in MCQA is performed by Multi-Head Self-Attention (MHSA) mechanisms rather than MLP layers. While MLPs tend to boost the probability of all valid answer symbols, a tiny, sparse group of attention heads—often just 1 to 4 per layer—is responsible for discriminating and promoting the correct symbol.
The Default Bias and Two-Stage Processing
When models encounter unusual symbols (like Q/Z/R/X), they undergo a two-stage process. In the middle layers, they initially default to assigning high probability to standard A/B/C/D labels. Only in the final layers do they "correct" this internal state to align with the specific symbols provided in the prompt.
Key Results and Training Milestones
In Olmo 7B, layer 24 was identified as the causal bottleneck; patching this single layer could flip the model's prediction. Furthermore, the authors identified a specific training milestone for this capability. For Olmo 7B, the ability to perform symbol binding on the "Colors" task jumped from near-random to nearly 100% between 80k and 100k training steps. Even after accuracy plateaus, the "logit gap"—the confidence margin—continues to grow with further training.
Conclusion and Future Directions
This work demonstrates that MCQA failures are often failures of "formatting" (symbol binding) rather than a lack of knowledge. By localizing these behaviors to specific attention heads, the study opens the door for "model surgery" to fix label biases. Future work will investigate how information is aggregated from earlier prompt tokens to feed these critical decision layers.
Summary updated