TL;DR
Ask2Decide is a new benchmark designed to evaluate whether LLM agents can identify and elicit the specific, missing information required to make optimal decisions in real-world planning tasks. The study reveals that while frontier LLMs are competent at modeling fully specified problems, they struggle significantly to ask the "right" questions, leading to plan failure rates as high as 98% when critical information is withheld.
What Problem Does This Paper Address?
In the real world, decision-making rarely happens with perfect information. A dispatcher might not know a driver needs to return early, or a hospital staffer might forget to mention a broken elevator. For an LLM agent to be truly useful, it must recognize when its knowledge base is incomplete and proactively ask for clarification.
Existing benchmarks fail to test this effectively for two reasons:
- Lack of Decision Value Measurement: They don't quantify how much a missing fact actually changes the outcome.
- Unrealistic User Simulation: Most interactive benchmarks use "default" LLM-simulated users that are overly polite, verbose, and helpful—unlike real humans who are often terse and may not offer the "correcting fact" even after a plan fails.
Methods
The authors frame planning as an optimization problem, allowing them to use mathematical solvers to provide ground-truth labels for the "value" of information.
- Minimax Regret ($\Delta$): The authors calculate the "regret" of each hidden fact—a formal measure of how much the optimal plan changes if that fact is known versus unknown.
- The "Twin" Control: To ensure agents aren't just asking about everything, the benchmark includes "twins"—facts that look identical in form to relevant ones (e.g., "Vehicle 2 must be back by 10:40") but have a regret of zero, meaning they don't actually change the optimal decision.
- CalOp (Calibrated Operator): The researchers conducted a human study with 19 participants to record how people actually respond to agents. They used this data to build CalOp, a simulated operator that mimics human terseness and the tendency to withhold information unless specifically asked.
Datasets
The Ask2Decide benchmark consists of 43 planning tasks across domains like routing, scheduling, and staffing. These are divided into five splits of increasing complexity:
- Easy, Medium, Hard: 24 textbook-style optimization problems (e.g., dial-a-ride, job shop scheduling).
- Frontier-A & B: 19 larger-scale problems with many named objects (flights, vessels, etc.) designed to push the limits of current models.
The tasks are built by taking existing optimization instances and systematically "hiding" operational facts (e.g., sudden accidents, human errors in records, or unwritten rules).
Key Insights and Ideas
- Information Acquisition is the Bottleneck: For many tasks, models perform perfectly when given all info up front. The failure happens almost entirely in the interaction phase—the models simply don't find out what they need to know.
- LLMs are "Broad" Askers, Not "Smart" Askers: Models tend to uncover relevant facts only slightly more often than irrelevant "twins" (a gap of only 5–11 percentage points). They lack the reasoning to prioritize questions that would most impact the decision.
- The "Rejection" Trap: In many benchmarks, a user saying "No, that plan is wrong" is enough for the LLM to guess the right answer. However, real humans often say a plan is wrong without explaining why. Ask2Decide shows that when simulators are calibrated to be as unhelpful as humans, LLM performance drops significantly.
Key Results
The authors evaluated four frontier models (including GPT-4 and Gemini variants) using the CalOp simulator:
- Infeasibility: When information was missing, the models' first plans were infeasible in 58% to 98% of tasks. When given full information, this dropped to 5% to 28%.
- The Asking Gap: The "asking gap" (the loss attributed specifically to the failure to acquire info) was consistently larger than the "all-info loss" (the loss from failing to model the problem).
- Recovery: While models could often "recover" and find a feasible plan after multiple rejections and revisions, this recovery often masked the fact that they never actually asked the right questions in the first place.
Limitations and Future Work
- Domain Specificity: The benchmark is currently focused on optimization and planning. Extending this to more open-ended reasoning tasks is a logical next step.
- Human Study Scale: While 79 sessions provided a good baseline for CalOp, a larger and more diverse pool of human operators could further refine the simulator.
- Model-Driven Questioning: The authors suggest that a "regret-guided" agent—one that uses a solver to rank which questions would most reduce decision uncertainty—could serve as a powerful baseline for future research.
Summary updated