TL;DR
Large Language Models (LLMs) are increasingly capable of solving multiple-choice questions (MCQs) using only the answer choices, bypassing the question entirely. This paper demonstrates that while test-time reasoning (TTR) boosts this "choices-only" success, it often does so through sophisticated strategies like inferring the missing question rather than just exploiting shallow dataset flaws.
What Problem Does This Paper Address?
In the world of NLP evaluation, "partial-input success" is usually seen as a red flag. If a model can answer a multiple-choice question without seeing the question, we typically assume the dataset is flawed—containing "shallow cues" or "shortcuts" that allow the model to cheat.
However, as LLMs gain advanced reasoning capabilities, the line between "cheating" and "intelligent guessing" blurs. This paper investigates whether test-time reasoning (TTR)—giving models "time to think" before answering—amplifies these shortcuts. More importantly, it uses qualitative analysis to see if the models' reasoning traces reveal truly shallow behavior or a more complex form of "strategic test-taking."
Methods
The researchers evaluated 12 high-performing LLMs (including Gemini 2.5, GPT-4.1/5, and Claude 3.5) across two primary settings:
- Full Input: The standard setup with both the question ($q$) and choices ($C$).
- Choices-Only: A "blind" setup where the model only sees the answer options ($C$).
They compared a Base prompt (direct answer) against a Reason prompt (eliciting step-by-step reasoning/TTR). To understand the how, they qualitatively coded 180 reasoning traces to categorize the strategies used. They also conducted "faithfulness" tests to ensure the reasoning traces actually reflected the models' decision-making process rather than being post-hoc justifications.
Datasets
The study utilized three popular benchmarks representing different difficulty levels:
- ARC: Grade-school science and commonsense reasoning.
- MMLU: College-level knowledge across 57 diverse topics.
- Super GPQA: A challenging benchmark testing graduate-level knowledge in fields like engineering and law.
Key Insights and Ideas
- Reasoning is a "Full-Input" Specialist: TTR provides a massive boost when the full question is present, but its impact on "choices-only" scenarios is much more modest. This suggests that the "scaling laws" of reasoning benefit genuine problem-solving more than they benefit "cheating."
- Abduction over Artifacts: Instead of just looking for statistical artifacts (like "pick the longest answer"), reasoning models often perform abductive reasoning. They look at the choices and work backward to infer what the question must have been. In 83% of successful "choices-only" cases on ARC, the model successfully guessed the original question's intent.
- The "Strategic Student" Analogy: The authors argue that LLMs behave like savvy human test-takers. When they don't have the question, they use elimination, recall related facts, and look for patterns. This shifts the perspective: "choices-only" success might be a sign of a model's broad knowledge and strategic thinking rather than just a dataset failure.
- Trace Faithfulness: The reasoning traces were found to be highly consistent with the final answers, meaning we can treat these traces as "soft evidence" for how the model is actually arriving at its conclusions.
Key Results
- Quantitative: TTR improved accuracy in 27 out of 36 "Full" settings but only 15 out of 36 "Choices-Only" settings. Interestingly, scaling the reasoning length (giving the model more "compute" at test time) only marginally improved choices-only performance.
- Qualitative: The authors identified five core strategies:
- FACT: Recalling external knowledge about the choices.
- ELIM: Eliminating inaccurate or unlikely options.
- PATTERNS: Identifying properties that group choices together.
- INFER Q: Guessing the missing question.
- SHALLOW: Using simple, non-knowledge-based cues (e.g., "this number looks messy").
- Predictors of Failure: Only "SHALLOW" cues and "INCONS" (inconsistent reasoning) were statistically significant predictors of the model getting the answer wrong.
Limitations and Future Work
The authors acknowledge that LLMs are highly sensitive to prompting; different prompts might yield higher "choices-only" scores, meaning their results might represent a lower bound. While they tested for faithfulness, it remains a difficult property to prove definitively in black-box models.
For future work, they propose using this style of trace analysis to help benchmark creators. By identifying which questions are solved via "SHALLOW" cues versus "FACT" or "ELIM" strategies, evaluators can filter out truly flawed items while keeping those that test genuine strategic reasoning. This could lead to a new generation of "shortcut-proof" benchmarks.