When we evaluate a human student's understanding of a topic, we often use multiple-choice questions (MCQs) as a convenient shorthand. They are easy to grade, objective, and scale perfectly. For years, the artificial intelligence community adopted the same strategy, using massive multiple-choice benchmarks to rank large language models (LLMs) on everything from common sense to advanced physics.
However, a profound disconnect has emerged: models that score near-perfectly on multiple-choice benchmarks frequently stumble when deployed in real-world, open-ended applications. This discrepancy is known as the generative evaluation gap. To understand why this gap exists, we must look at how multiple-choice formats fundamentally alter the nature of the task presented to a language model.
Recognition versus Generation
The core reason multiple-choice benchmarks mislead us lies in the difference between recognition and generation.
When an LLM generates text in the wild, it navigates an unbounded search space. At each step, it must select the next token from a vocabulary of tens of thousands of possibilities, balancing logic, syntax, and context. In contrast, a multiple-choice format collapses this infinite search space into a tiny, closed set of options (typically A, B, C, or D). The model does not need to construct a coherent path to an answer; it only needs to compute which of the provided options has the highest relative token probability.
As demonstrated by Li et al. in Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs?, a model's success in selecting the correct answer from a list correlates poorly with its ability to generate an accurate, long-form response for the exact same question. By restricting the output space, MCQs measure narrow recognition skills rather than the constructive reasoning required for real-world utility.
Generative Setting (Unbounded Space)
[Prompt] ---> [Model Reasoning] ---> Token 1 ---> Token 2 ---> ... (Infinite Paths)
Multiple-Choice Setting (Constrained Space)
[Prompt] + [Options A, B, C, D] ---> Compare P(A), P(B), P(C), P(D) ---> Select Highest
The Mechanics of Multiple-Choice Shortcuts
Because multiple-choice formats simplify the evaluation environment, models can exploit several "shortcuts" to arrive at the correct answer without genuine underlying comprehension.
Heuristic Elimination and Guessing
In an open-ended setting, a mathematical or logical error at any step will derail the final output. In a multiple-choice setting, however, a model can make a reasoning error but still luck into the correct answer by eliminating options that are grammatically or logically absurd.
Fang et al., in their work on MathOdyssey, point out that fixed options allow models to bypass generative reasoning entirely through heuristics. This inflation of capability is even more pronounced in elite problem-solving. As argued by Qin et al. in RIMO, small answer spaces allow models to achieve high scores through lucky guessing or by exploiting a limited search space, masking their true analytical limitations.
Information Leakage
The options in a multiple-choice test do not just restrict the answer; they actively leak information about what the answer should look like. In SciEx, Dinh et al. observe that the presence of distractor options can inadvertently tip off the model to the correct answer, leading to an overestimation of its scientific reasoning.
This "crutch" is equally problematic for multimodal models. When evaluating vision-language models on biological images in VLM4Bio, Altıntaş et al. discovered that models struggle significantly with open-ended species classification once they can no longer rely on the process of elimination provided by multiple-choice options.
Superficial Biases
Multiple-choice formats introduce artificial vulnerabilities that have nothing to do with actual knowledge. Models suffer from severe positional biases (such as favoring the first option) and extreme sensitivity to how the options are framed.
By mapping over 250 million model predictions in DOVE, Habba et al. demonstrated that switching from "A, B, C" to "1, 2, 3" can cause a model's accuracy to swing wildly. Because multiple-choice evaluations are uniquely sensitive to these formatting artifacts, they often measure a model's formatting alignment rather than its true capabilities.
The Disconnect in Complex Domains
The generative evaluation gap becomes a critical bottleneck when assessing advanced reasoning, safety, and nuanced human values.
Faithfulness and Reasoning Trajectories
For modern large reasoning models that generate internal "thinking drafts" before answering, multiple-choice testing obscures whether the model is actually following its own logic. Xiong et al. explored this in Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models, revealing that the structured nature of label selection simplifies the assessment of consistency, thereby overlooking complex logical disconnects in free-form reasoning.
Similarly, Payoungkhamdee et al. found in An Empirical Study of Multilingual Reasoning Distillation for Question Answering that reasoning capabilities distilled into smaller models based on multiple-choice performance frequently fail to generalize to open-ended generative tasks like creative writing or strategic planning.
Constructive and Social Nuance
In fields like mathematics, law, and ethics, true mastery is constructive. In MathConstruct, Balunovic et al. argue that short-form or multiple-choice answers fail to capture the constructive process required to build complex mathematical objects like matrices or graphs.
When it comes to human interaction, the gap is even wider. Social communication and morality are rarely binary choices; they exist in a gray area of probability and context. Wu et al. point out in Rethinking Pragmatics in Large Language Models that a model might easily identify the socially appropriate response in a constrained multiple-choice test, yet remain entirely unable to generate appropriate pragmatic language in a fluid, free-form dialogue. The same limitation applies to capturing moral failures, where Yu et al. note in CMoralEval that standardized multiple-choice benchmarks fail to reflect how a model will navigate real-world ethical dilemmas in open-ended text generation.
Designing Better Examiners
To close the generative evaluation gap, the research community is shifting toward evaluation paradigms that preserve the open-ended nature of language generation while maintaining rigorous measurement.
One approach is to evaluate models probabilistically across open-ended distributions. In Every Answer Matters: Evaluating Commonsense with Probabilistic Measures, Boratko et al. introduce a framework where models must generate multiple open-ended answers, measuring how well the model's entire answer distribution matches human consensus using KL divergence. This removes the shortcut of a single correct option and forces the model to capture the nuanced, probabilistic nature of common sense.
Another direction replaces static, multiple-choice datasets with dynamic, generative testing environments. Bai et al. propose a decentralized "peer-examination" system in Benchmarking Foundation Models with Language-Model-as-an-Examiner, where an LLM acts as a dynamic examiner—generating open-ended questions and evaluating responses in a reference-free manner to eliminate data leakage and recognition shortcuts.
Finally, to ensure that structured evaluation can coexist with open-ended generation, Sun introduces StructEval, a framework that expands structured testing methods to non-constrained response formats across multiple cognitive levels. By testing models on their ability to construct explanations rather than select labels, these new methodologies aim to provide a more faithful reflection of how AI systems will perform in the real world.