From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan ZhouarXiv 2026 · arXiv

TL;DR

Ask2Decide is a new benchmark designed to evaluate whether LLM agents can identify and elicit the specific, missing information required to make optimal decisions in real-world planning tasks. The study reveals that while frontier LLMs are competent at modeling fully specified problems, they struggle significantly to ask the "right" questions, leading to plan failure rates as high as 98% when critical information is withheld.

What Problem Does This Paper Address?

In the real world, decision-making rarely happens with perfect information. A dispatcher might not know a driver needs to return early, or a hospital staffer might forget to mention a broken elevator. For an LLM agent to be truly useful, it must recognize when its knowledge base is incomplete and proactively ask for clarification.

Existing benchmarks fail to test this effectively for two reasons:

  1. Lack of Decision Value Measurement: They don't quantify how much a missing fact actually changes the outcome.
  2. Unrealistic User Simulation: Most interactive benchmarks use "default" LLM-simulated users that are overly polite, verbose, and helpful—unlike real humans who are often terse and may not offer the "correcting fact" even after a plan fails.

Methods

The authors frame planning as an optimization problem, allowing them to use mathematical solvers to provide ground-truth labels for the "value" of information.

  • Minimax Regret ($\Delta$): The authors calculate the "regret" of each hidden fact—a formal measure of how much the optimal plan changes if that fact is known versus unknown.
  • The "Twin" Control: To ensure agents aren't just asking about everything, the benchmark includes "twins"—facts that look identical in form to relevant ones (e.g., "Vehicle 2 must be back by 10:40") but have a regret of zero, meaning they don't actually change the optimal decision.
  • CalOp (Calibrated Operator): The researchers conducted a human study with 19 participants to record how people actually respond to agents. They used this data to build CalOp, a simulated operator that mimics human terseness and the tendency to withhold information unless specifically asked.
Figure 1: Overview of Ask2Decide on hard1_dialaride. (a) The agent gets a brief and the records; the operator holds today’s notes, each labelled by the solver with its regret ∆. The twin H5 has the same form as H2 but leaves the best plan unchanged. (b) The operator answers only questions about a specific object, and the agent’s first plan is scored on the true problem. Replies are constructed for illustration.

Datasets

The Ask2Decide benchmark consists of 43 planning tasks across domains like routing, scheduling, and staffing. These are divided into five splits of increasing complexity:

  • Easy, Medium, Hard: 24 textbook-style optimization problems (e.g., dial-a-ride, job shop scheduling).
  • Frontier-A & B: 19 larger-scale problems with many named objects (flights, vessels, etc.) designed to push the limits of current models.

The tasks are built by taking existing optimization instances and systematically "hiding" operational facts (e.g., sudden accidents, human errors in records, or unwritten rules).

Figure 2: How a task is built (dial-a-ride example). Steps 1–2 are written by the task author, or by a script for the frontier splits; steps 3–4 are run by the solver. Below each step: the design criterion it secures.

Key Insights and Ideas

  • Information Acquisition is the Bottleneck: For many tasks, models perform perfectly when given all info up front. The failure happens almost entirely in the interaction phase—the models simply don't find out what they need to know.
  • LLMs are "Broad" Askers, Not "Smart" Askers: Models tend to uncover relevant facts only slightly more often than irrelevant "twins" (a gap of only 5–11 percentage points). They lack the reasoning to prioritize questions that would most impact the decision.
  • The "Rejection" Trap: In many benchmarks, a user saying "No, that plan is wrong" is enough for the LLM to guess the right answer. However, real humans often say a plan is wrong without explaining why. Ask2Decide shows that when simulators are calibrated to be as unhelpful as humans, LLM performance drops significantly.

Key Results

The authors evaluated four frontier models (including GPT-4 and Gemini variants) using the CalOp simulator:

  • Infeasibility: When information was missing, the models' first plans were infeasible in 58% to 98% of tasks. When given full information, this dropped to 5% to 28%.
  • The Asking Gap: The "asking gap" (the loss attributed specifically to the failure to acquire info) was consistently larger than the "all-info loss" (the loss from failing to model the problem).
  • Recovery: While models could often "recover" and find a feasible plan after multiple rejections and revisions, this recovery often masked the fact that they never actually asked the right questions in the first place.

Limitations and Future Work

  • Domain Specificity: The benchmark is currently focused on optimization and planning. Extending this to more open-ended reasoning tasks is a logical next step.
  • Human Study Scale: While 79 sessions provided a good baseline for CalOp, a larger and more diverse pool of human operators could further refine the simulator.
  • Model-Driven Questioning: The authors suggest that a "regret-guided" agent—one that uses a solver to rank which questions would most reduce decision uncertainty—could serve as a powerful baseline for future research.

Summary updated

Abstract

LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiring agents to ask for clarification. Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested. We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions. An operator holds facts absent from the agent's request and records, and optimization solvers determine which facts change the optimal plan. We also include similarly worded facts that leave the optimal plan unchanged to test whether agents distinguish useful information from irrelevant details. We collect 79 interaction sessions from 19 participants and find that, after an incorrect plan, human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%). Using these interactions, we calibrate a simulated operator that more closely matches human reply patterns. Evaluating four frontier LLMs, we find that their initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront. Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%). These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. This is ongoing work and will be updated. The date of this version is shown on the first page.
BibTeX
@misc{tang2026from,
  author = {Yuhan Tang and Ke Fan and Xinkai Zou and Baichuan Mo and Kangxin Cui and Ao Qu and Ou Zheng and Jiangbo Yu and Haris N. Koutsopoulos and Jinhua Zhao and Yuan Zhou},
  title = {{From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings}},
  year = {2026},
  eprint = {2611.13015},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13015},
}
Figure