TL;DR
Large Language Models (LLMs) often provide a single, confident answer to ambiguous questions, masking the fact that multiple interpretations exist. This paper introduces a framework to detect this ambiguity by applying "semantics-blind" perturbations—tiny changes to the input that preserve the original ambiguity—and measuring how much the model's answers fluctuate.
What Problem Does This Paper Address?
When you ask an LLM an underspecified question like "When did the team win the championship?", the model often "silently commits" to one interpretation (e.g., the 2024 Kansas City Chiefs) and gives a fluent, confident answer. This is a major hurdle for trustworthy AI; ideally, the model should recognize the ambiguity and ask for clarification.
Existing methods for detecting this uncertainty usually involve asking the LLM to rephrase the question. However, the authors identify a "subtle flaw" here: to rephrase an ambiguous question, the LLM must first understand it, and in doing so, it often accidentally resolves the ambiguity. By the time you measure the model's consistency across rephrasings, the ambiguity signal has already been erased.
Methods
The authors propose a two-stage objective: detect ambiguity first, then mitigate it. Their core principle is that detection should be "semantics-blind"—it shouldn't involve the model trying to "understand" or rewrite the question's meaning.
They instantiate this through two types of detectors:
- White-box Detector: Uses internal model signals (like gradients or the "Logit Lens") to identify which specific tokens are most influential to the output. It then applies low-capacity interventions like masking or deleting those specific tokens.
- Black-box Detector: Uses a library of rule-based, surface-level transformations that don't require internal access. These include swapping synonyms (via WordNet), expanding/contracting abbreviations (e.g., "Who's" vs "Who is"), or adding generic prefixes (e.g., "Hey so...").
After perturbing the input, they sample multiple answers and compute a Pairwise Semantic Agreement Matrix. If the answers vary significantly across these tiny surface changes, the question is flagged as likely ambiguous.
Datasets
The framework was evaluated across four English QA benchmarks with explicit ambiguity annotations:
- ABG-CoQA: Conversational QA with ambiguity labels.
- AmbigQA-w-Ctx: Open-domain QA paired with Wikipedia context.
- AQuA: Visual QA involving referential ambiguity (e.g., multiple objects the question could refer to).
- VQ-FA: Another visual QA dataset focused on focus ambiguity.
Key Insights and Ideas
- Paraphrasing is a "High-Capacity" Trap: The most striking finding is that using an LLM to paraphrase a question actually reduces the measured uncertainty. Because the LLM picks one interpretation during the rewrite, it hides the original ambiguity from the detection process.
- The Information Bottleneck Principle: The authors frame their approach as a trade-off: discard information about the input's surface form while strictly preserving the ambiguity label.
- Ambiguity is Distributed: No single perturbation rule is a silver bullet. The best performance comes from aggregating the signal across a diverse family of small rules (synonyms, prefixes, etc.).
Key Results
- Superior Detection: The perturbation-based methods consistently outperformed standard baselines (like Semantic Entropy or Direct Prompting) across Qwen, Llama, and Gemma model families. In some cases, they achieved an absolute improvement of up to 0.07 in AUROC.
- Visual QA Gains: The improvement was most pronounced in visual QA (AQuA), where referential ambiguity is notoriously hard to surface from output variance alone.
- Effective Mitigation: The authors introduced UQ-Augmented Re-asking (UAR). By using the detector's output to re-prompt the model with the evidence of its own instability, they improved answer correctness by +8.15 to +18.07 points in Exact Match scores.
Limitations and Future Work
- Language Dependency: The black-box rule library is currently built for English. Applying this to other languages would require reconstructing these lexical and orthographic rules.
- Access Requirements: The white-box detector requires access to model internals (gradients/hidden states), which isn't possible for closed-source APIs like GPT-4.
- Silent Commitment: The authors note that models sometimes commit so strongly to a single salient referent that even surface perturbations fail to expose the underlying ambiguity (a source of false negatives).
Future work may involve automating the discovery of "semantics-blind" rules for new languages or domains.
Summary updated