Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs

Tiejin Chen, Xiaoou Liu, Michael Clifford, Ahmadreza Moradipari, Yitao Chen, Kyungtae Han, Nejib Ammar, Hua WeiarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) often provide a single, confident answer to ambiguous questions, masking the fact that multiple interpretations exist. This paper introduces a framework to detect this ambiguity by applying "semantics-blind" perturbations—tiny changes to the input that preserve the original ambiguity—and measuring how much the model's answers fluctuate.

What Problem Does This Paper Address?

When you ask an LLM an underspecified question like "When did the team win the championship?", the model often "silently commits" to one interpretation (e.g., the 2024 Kansas City Chiefs) and gives a fluent, confident answer. This is a major hurdle for trustworthy AI; ideally, the model should recognize the ambiguity and ask for clarification.

Existing methods for detecting this uncertainty usually involve asking the LLM to rephrase the question. However, the authors identify a "subtle flaw" here: to rephrase an ambiguous question, the LLM must first understand it, and in doing so, it often accidentally resolves the ambiguity. By the time you measure the model's consistency across rephrasings, the ambiguity signal has already been erased.

Methods

The authors propose a two-stage objective: detect ambiguity first, then mitigate it. Their core principle is that detection should be "semantics-blind"—it shouldn't involve the model trying to "understand" or rewrite the question's meaning.

Figure 1: Overview of the proposed ambiguity detection framework. Given an input question and perturbation principles, the method generates label-preserving, low-capacity, and semantics-blind perturbations via white-box token-importance methods and black-box rule-based transformations. The perturbed questions are then sampled to obtain multiple model answers, whose pairwise semantic agreement is used to compute an ambiguity signal: low agreement indicates likely ambiguity, while high agreement suggests the question is likely unambiguous.

They instantiate this through two types of detectors:

  1. White-box Detector: Uses internal model signals (like gradients or the "Logit Lens") to identify which specific tokens are most influential to the output. It then applies low-capacity interventions like masking or deleting those specific tokens.
  2. Black-box Detector: Uses a library of rule-based, surface-level transformations that don't require internal access. These include swapping synonyms (via WordNet), expanding/contracting abbreviations (e.g., "Who's" vs "Who is"), or adding generic prefixes (e.g., "Hey so...").

After perturbing the input, they sample multiple answers and compute a Pairwise Semantic Agreement Matrix. If the answers vary significantly across these tiny surface changes, the question is flagged as likely ambiguous.

Datasets

The framework was evaluated across four English QA benchmarks with explicit ambiguity annotations:

  • ABG-CoQA: Conversational QA with ambiguity labels.
  • AmbigQA-w-Ctx: Open-domain QA paired with Wikipedia context.
  • AQuA: Visual QA involving referential ambiguity (e.g., multiple objects the question could refer to).
  • VQ-FA: Another visual QA dataset focused on focus ambiguity.

Key Insights and Ideas

  • Paraphrasing is a "High-Capacity" Trap: The most striking finding is that using an LLM to paraphrase a question actually reduces the measured uncertainty. Because the LLM picks one interpretation during the rewrite, it hides the original ambiguity from the detection process.
  • The Information Bottleneck Principle: The authors frame their approach as a trade-off: discard information about the input's surface form while strictly preserving the ambiguity label.
  • Ambiguity is Distributed: No single perturbation rule is a silver bullet. The best performance comes from aggregating the signal across a diverse family of small rules (synonyms, prefixes, etc.).
Figure 3: AUROC comparison across individual per- turbation rules and their combined cross-variant setting. While several single perturbations improve over the orig- inal input, combining all perturbation variants achieves the strongest ambiguity detection performance.

Key Results

  • Superior Detection: The perturbation-based methods consistently outperformed standard baselines (like Semantic Entropy or Direct Prompting) across Qwen, Llama, and Gemma model families. In some cases, they achieved an absolute improvement of up to 0.07 in AUROC.
  • Visual QA Gains: The improvement was most pronounced in visual QA (AQuA), where referential ambiguity is notoriously hard to surface from output variance alone.
  • Effective Mitigation: The authors introduced UQ-Augmented Re-asking (UAR). By using the detector's output to re-prompt the model with the evidence of its own instability, they improved answer correctness by +8.15 to +18.07 points in Exact Match scores.
Figure 4: Comparison of performance and absolute value of uncertainty on ambiguous questions from AquA. The results show that directly using LLM- paraphrasing can actually reduce the uncertainty.

Limitations and Future Work

  • Language Dependency: The black-box rule library is currently built for English. Applying this to other languages would require reconstructing these lexical and orthographic rules.
  • Access Requirements: The white-box detector requires access to model internals (gradients/hidden states), which isn't possible for closed-source APIs like GPT-4.
  • Silent Commitment: The authors note that models sometimes commit so strongly to a single salient referent that even surface perturbations fail to expose the underlying ambiguity (a source of false negatives).

Future work may involve automating the discovery of "semantics-blind" rules for new languages or domains.

Summary updated

Abstract

Large language models (LLMs) often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt admits multiple valid interpretations, the key question is whether the model should answer directly at all. Existing input uncertainty methods often rely on LLM-generated statements or clarifications, but such transformations may already disambiguate the input before the ambiguity is measured. We propose a label-preserving perturbation framework for input ambiguity detection in which perturbations remain close to the original input while preserving its label. We instantiate this principle with two detectors: a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals. Experiments across four datasets in open-domain, conversational, and visual QA show that our methods outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. Our results highlight the importance of detecting ambiguity before attempting to disambiguate it. Our code is available \href{https://github.com/Xiao0o0o/input-ambiguity-detection}{\textcolor{purple}{here}}.
BibTeX
@misc{chen2026detect,
  author = {Tiejin Chen and Xiaoou Liu and Michael Clifford and Ahmadreza Moradipari and Yitao Chen and Kyungtae Han and Nejib Ammar and Hua Wei},
  title = {{Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs}},
  year = {2026},
  eprint = {2610.36353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36353},
}
Figure