Automated Generation of Plausible Multiple Choice Distractors

40 concepts38 papers2 proposals

Evaluating modern language and vision models is notoriously difficult. Open-ended generation is hard to grade automatically without relying on brittle text-matching metrics or expensive, shifting "LLM-as-a-judge" setups. Converting these tasks into a multiple-choice format offers a clean, reproducible alternative.

However, a multiple-choice benchmark is only as good as its incorrect options—the distractors. If the distractors are random or superficial, models can easily exploit linguistic shortcuts, frequency biases, or keyword overlaps to guess the correct answer without true comprehension, as noted by Zhu et al.. To build rigorous benchmarks, researchers have turned to automated distractor generation: using language models to systematically engineer alternatives that are contextually, semantically, and syntactically plausible, yet logically or factually incorrect.

The Plausibility Gap

Why do simple negative samples fail? Consider a question about a complex visual scene or a piece of code. If a distractor contains out-of-distribution vocabulary or lacks the stylistic nuances of the ground truth, an evaluator model can reject it purely on surface-level features.

To close this gap, automated counterfactual generation utilizes large-scale language models to produce incorrect continuations that mimic the exact linguistic style and structure of the correct answer. As demonstrated by Zellers et al. with the SWAG dataset, this forces evaluating models to move past stylistic tells and actually reason about the underlying scenario. When distractors share the same distribution as the true response, the task transitions from simple pattern recognition to deep semantic analysis.

Leveraging Internal Model Knowledge

How do we systematically generate an alternative that is wrong but highly convincing? One elegant approach is to turn a model's own internal probabilities against it.

[Target Question] ──> [Closed-Book LM] ──> [Topically Relevant but Factually Flawed Foil]
                                                    │
                                                    ▼
                                        (High-Plausibility Distractor)

Servedio et al. leverage perplexity-guided sampling to identify challenging evaluation data. By selecting incorrect statements that a language model assigns a high probability to within its own distribution, the authors capture realistic hallucinations. Because the model's internal states find these sequences highly plausible, they serve as the ultimate test of factuality.

Another method, introduced by Lelkes et al., forces a language model to answer questions in a "closed-book" setting without access to the source text. The model naturally generates answers that are topically relevant and highly authoritative in tone, but factually incorrect for the specific context provided. These model-generated foils mimic human-like errors so closely that they effectively expose whether an evaluated model is truly reading the context or just guessing based on general knowledge.

Semantic Perturbation and Component Swapping

Instead of generating distractors from scratch, another powerful mechanism mutates the ground truth directly. This ensures perfect stylistic alignment while surgically altering the logic.

  • Word Masking and Perturbation: An used masked language models to perturb specific words within a ground-truth response. This open-ended discovery mechanism uncovers subtle model biases by creating alternatives that differ by only a few key terms.
  • Component Swapping: In domains like figurative language, Qu et al. showed that benchmarks can swap core components or replace words with synonyms. This tests whether a model understands deep metaphorical connections or is simply anchored to surface-level vocabulary.
  • Embedding-Based Retrieval: For complex visual reasoning, Yang et al. combined linguistic perturbations with embedding-based retrieval. By identifying distractors that are mathematically close in vector space but contextually irrelevant, they prevent models from relying on simple keyword matching.

Adversarial and Multi-Agent Refinement

To maximize the difficulty of these options, distractor generation can be framed as an active adversarial game or an agentic pipeline.

Khot et al. introduced a multi-adversary approach that explicitly selects distractors designed to fool high-performing language models and retrieval systems. If a state-of-the-art model or an information retrieval heuristic ranks an incorrect option highly, that option is selected as a distractor, guaranteeing that the final multiple-choice question cannot be solved via shallow heuristics.

                       ┌─────────────────────────┐
                       │     Proposer Agent      │
                       │ (Simulates Model Flaws) │
                       └────────────┬────────────┘
                                    │
                                    ▼
                        [Candidate Distractors]
                                    │
                                    ▼
                       ┌─────────────────────────┐
                       │     Reviewer Agent      │
                       │ (Filters & Sharpens)    │
                       └────────────┬────────────┘
                                    │
                                    ▼
                        [High-Quality Benchmark]

Taking this a step further, Su proposed an agentic framework called AutoConverter. This architecture splits the task between a proposer agent and a reviewer agent. The proposer agent explicitly simulates known vision-language model failure modes—such as visual misinterpretations or reasoning slips—to draft options. The reviewer agent then filters and sharpens these options, ensuring they remain definitively incorrect while maximizing their deceptive potential.

From Evaluation to Calibration

While automated distractor generation is primarily used to build rigorous benchmarks across specialized domains—such as code understanding (Nguyen et al.), code reviews (Lin et al.), long video grounding (Di et al.), and audio reasoning (Sakshi et al.)—it also serves a powerful secondary purpose: improving model honesty.

Chhikara demonstrated that presenting models with multiple-choice formats containing highly plausible distractors drastically improves their calibration. When forced to navigate a field of near-factual options rather than generating open-ended text, models show a massive reduction in Expected Calibration Error (ECE). Navigating these automated foils forces models to evaluate their own certainty, transforming a tool for benchmark generation into a mechanism for mitigating overconfidence.

Key Foundational Work

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018) — contributed 1 concept elements to this direction.
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets (2018) — contributed 1 concept elements to this direction.
Visual7W: Grounded Question Answering in Images (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_distractor_generation_a3a0c17d,
  title = {automated distractor generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-distractor-generation/rdr_a3a0c17d3f8446dda64a3d8e31ff9de6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_distractor_generation_a3a0c17d_v1,
  title = {automated distractor generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-distractor-generation/rdr_a3a0c17d3f8446dda64a3d8e31ff9de6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake