Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models

Avik Dutta, Aayush Kumar, Zahra Ashktorab, Hosein Hasanbeig, Austin Z. Henley, Gustavo SoaresarXiv 2026 · arXiv

TL;DR

Researchers at Microsoft have developed a new framework to evaluate how Large Language Models (LLMs) resolve ambiguity during multi-turn conversations. By using a "preference-grounded" user simulator and tracking specific "dissatisfaction signals," they’ve identified a critical trade-off between models that guess too much and those that ask too many annoying questions, ultimately providing a path toward more efficient, "Pareto-optimal" AI assistants.

What Problem Does This Paper Address?

Real-world user queries are notoriously underspecified. If you ask an AI to "list the top-performing senior staff," does "senior" mean age, job title, or tenure? Does "top-performing" refer to sales figures or manager feedback?

Currently, we evaluate LLMs on ambiguity using static metrics that check if a model can identify all possible interpretations. However, this doesn't reflect how we actually use AI: in a conversation. In a multi-turn setting, models often fall into two traps:

  1. The Cowboy Approach: Making aggressive assumptions that lead to incorrect results.
  2. The Interrogator Approach: Asking repetitive, irrelevant, or "over-clarifying" questions that frustrate the user.

Existing evaluation frameworks often rely on other LLMs to act as "users," but these simulators are prone to hallucinating preferences or "leaking" the right answer too early, making the evaluation unreliable.

Methods

The authors introduce two major innovations to fix interactive evaluation:

  1. Preference-Grounded User Simulator: Instead of a free-form LLM, this simulator is strictly constrained by a User Disambiguation Preference Set ($P$). This set contains the minimal specifications needed to uniquely identify the user's intent. The simulator only reveals information if the model asks a relevant question or makes a wrong assumption. This ensures the evaluation is reproducible and free from "information leakage."
  2. Dissatisfaction Signals: Moving beyond simple "task accuracy," the paper defines three metrics to measure interaction quality:
  3. Incorrect Assumptions: When the model acts on a guess that violates the user's hidden preferences.
  4. Repeating Clarifications: Asking a question the user has already answered.
  5. Irrelevant Clarifications: Asking questions that don't actually help narrow down the specific ambiguity at hand.

Finally, they propose DisPO (Dissatisfaction-guided Prompt Optimization), a method to iteratively refine system prompts to minimize these three signals.

Datasets

The framework was tested across three diverse benchmarks to ensure broad applicability:

  • AmbigQA: Open-domain question answering (based on NaturalQuestions).
  • Ambrosia: Text-to-SQL semantic parsing (handling scope and vagueness in database queries).
  • nvBench 2.0: Text-to-visualization (generating charts from tabular data).

Key Insights and Ideas

  • The Ambiguity Trade-off: There is a clear Pareto frontier. Models that seek more clarification naturally make fewer incorrect assumptions but suffer from higher rates of irrelevant or repetitive questions.
  • The "Clean Interaction Rate" (CIR): The authors propose CIR as a gold-standard metric—it measures the percentage of interactions that reach the correct answer without triggering any of the three dissatisfaction signals.
  • Open-Weight vs. Proprietary: Closed-weight models (like GPT-5.4 and Claude-Opus-4.6) are significantly better at "assuming wisely." Open-weight models (like Phi-4 and DeepSeek-R1) tend to struggle with inferring intent, often requiring more turns to reach the same goal.
  • Simulator Robustness: The preference-grounded approach proved much more stable than standard LLM-only simulators, which often "cheated" by giving the model hints or failing to notice when the model made a wrong assumption.

Key Results

  • Optimization Works: Using DisPO to optimize system prompts reduced overall error rates by up to 7.65%.
  • Pareto Optimality: Optimized models achieved a superior balance, asking only necessary questions while maintaining high accuracy.
  • Efficiency: Models with shorter conversation lengths generally achieved higher CIR, proving that "asking precisely" is the key to conversational efficiency. For example, on the AmbigQA dataset, GPT-5.4 achieved a 31.84% CIR, while the optimized DisPO version pushed this significantly higher.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Preference Quality: The framework is only as good as the "preference sets" created for the benchmarks. Generating these for unbounded, real-world queries remains a challenge.
  • Subjectivity: Defining what constitutes a "valid" or "popular" interpretation involves some level of subjectivity.
  • Single-Turn Limitations: The study focuses on multi-turn efficiency and does not explore models that provide multiple interpretations simultaneously in a single response.

Future work may involve extending this to more complex, "unbounded" ambiguity spaces where the number of interpretations isn't easily countable.

Summary updated

Abstract

Ambiguity is a core challenge in human--AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals --- incorrect assumptions, repeating clarifications, and irrelevant clarifications --- that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.
BibTeX
@misc{dutta2026ask,
  author = {Avik Dutta and Aayush Kumar and Zahra Ashktorab and Hosein Hasanbeig and Austin Z. Henley and Gustavo Soares},
  title = {{Ask Precisely, Assume Wisely: Interactive Evaluation of Ambiguity Resolution in Large Language Models}},
  year = {2026},
  eprint = {2611.18905},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18905},
}
Figure