TL;DR
While modern LLM agents excel at executing tool calls, they frequently fail at understanding real people. UserBench is a new Gymnasium-based environment that evaluates agents in interactive, multi-turn travel planning where users provide vague goals and reveal implicit preferences incrementally. The benchmark reveals a stark gap: state-of-the-art models actively uncover fewer than 30% of user preferences and fully align with user intent only ~20% of the time.
What Problem Does This Paper Address?
Most existing agent benchmarks evaluate whether an LLM can parse a clear prompt, select APIs, and return an answer. In the real world, human communication rarely works this way. Human instructions exhibit three core challenges:
- Underspecification: Users initiate tasks before fully formulating their requirements.
- Incrementality: True goals and constraints emerge dynamically across the conversation.
- Indirectness: Users express constraints subtly or implicitly (e.g., saying "I get restless with long layovers" rather than explicitly requesting direct flights).
Existing agent benchmarks assume fully specified prompts or treat users as static query generators. As a result, agents often make premature, unaligned guesses rather than acting as collaborative partners.
Methods
UserBench casts user-centric alignment as an interactive reinforcement learning environment built on the standard Gymnasium framework:
- Interactive Simulated User: Powered by GPT-4o, the simulated user holds hidden travel preferences and budget constraints. It reveals them indirectly when the agent asks precise, targeted questions (active elicitation) or after several stalled turns (passive elicitation).
- Agent Action Space: Agents interact via a unified tool interface supporting three actions:
search: Queries a structured database for mock options (flights, hotels, restaurants, car rentals, apartments).action: Communicates directly with the user to ask clarifying questions.answer: Submits a final recommendation ID evaluated against ground-truth preferences and budget utility.- Realistic Distractors: Search returns a mix of correct options (satisfying all preferences), wrong options (violating at least one preference), and noise options (irrelevant or corrupted results).
- Dual Evaluation Settings:
- Single-choice: The agent must recommend exactly one option per aspect.
- Multi-choice: The agent can output multiple options, scored by the best choice provided.
Datasets
The authors curated a benchmark consisting of 4,000+ travel planning scenarios (471 test instances, 2,651 reserved for training):
- Covers 5 core aspects (hotel, restaurant, flight, rental car, apartment).
- Includes 82 distinct user preferences across 24 categories, paired with 902 carefully validated implicit elicitation statements.
- Categorized into three difficulty tiers (Easy, Medium, Hard) depending on the number and distribution of preferences per itinerary.
Key Insights and Ideas
- Tool use is solved; conversation is not: Top models achieved >80% validity on database search syntax, but their valid clarifying action rates hovered below 30–40%.
- More dialogue turns do not imply better alignment: Giving agents up to 20 or 30 turns did not consistently improve performance. Without explicit dialogue planning, agents frequently devolved into repetitive loops or off-topic chatter.
- Preference density per aspect is the real bottleneck: Spreading 4 preferences across 4 aspects (e.g., one for flight, one for hotel, etc.) yielded much higher scores than concentrating 4 preferences into a single aspect, showing that local constraint satisfaction quickly overloads agent reasoning.
- The "Guessing" illusion in multi-choice settings: Agents scored ~40% higher in multi-choice mode compared to single-choice mode. However, their active preference elicitation rates stayed flat or declined—meaning the improvement came from brute-force guessing rather than genuine understanding.
- Elicitation vs. Integration gap: Claude-4-Sonnet elicited the highest percentage of preferences (over 34% actively/passively), yet GPT-4o achieved higher final task scores. Knowing what the user wants does not automatically translate into choosing the optimal candidate.
Key Results
Under the primary single-choice setting across leading closed- and open-source models:
- Overall Scores: GPT-4o achieved the highest score (0.329), followed closely by Gemini-2.5-Pro (0.317) and Claude-4-Sonnet (0.307). Open-source models like DeepSeek-V3 (0.210) and Qwen3-32B (0.206) lagged behind.
- Correct Option Rates: Even top-tier models selected user-aligned options only 31% to 36% of the time (
Correct Exist Rate), and the single best cost-effective option only 20% to 26% of the time (Best Exist Rate). - Active Elicitation: Top performers proactively uncovered less than 27% of hidden user preferences (GPT-4o: 24.06% active; Gemini-2.5-Pro: 23.85% active; Claude-4-Sonnet: 26.31% active). Smaller models (Llama-3.1-8B, Qwen3-8B) actively elicited under 12%.
Limitations and Future Work
- Domain Specificity: The environment currently centers on travel planning; extending it to domains like coding assistance, customer service, or healthcare remains open.
- Simulated User Artifacts: While GPT-4o models human conversational nuances reasonably well, simulated users may exhibit predictability that differs from live human subjects.
- Balancing Interaction Efficiency and Probing: Models often guess prematurely to minimize turn count. Future work involves using UserBench as a training ground for multi-turn reinforcement learning (RL), designing reward schedules that penalize unprompted guessing while rewarding systematic intent elicitation.
Summary updated