UserBench: An Interactive Gym Environment for User-Centric Agents

Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang

TL;DR

While modern LLM agents excel at executing tool calls, they frequently fail at understanding real people. UserBench is a new Gymnasium-based environment that evaluates agents in interactive, multi-turn travel planning where users provide vague goals and reveal implicit preferences incrementally. The benchmark reveals a stark gap: state-of-the-art models actively uncover fewer than 30% of user preferences and fully align with user intent only ~20% of the time.

What Problem Does This Paper Address?

Most existing agent benchmarks evaluate whether an LLM can parse a clear prompt, select APIs, and return an answer. In the real world, human communication rarely works this way. Human instructions exhibit three core challenges:

  1. Underspecification: Users initiate tasks before fully formulating their requirements.
  2. Incrementality: True goals and constraints emerge dynamically across the conversation.
  3. Indirectness: Users express constraints subtly or implicitly (e.g., saying "I get restless with long layovers" rather than explicitly requesting direct flights).

Existing agent benchmarks assume fully specified prompts or treat users as static query generators. As a result, agents often make premature, unaligned guesses rather than acting as collaborative partners.

Methods

UserBench casts user-centric alignment as an interactive reinforcement learning environment built on the standard Gymnasium framework:

  • Interactive Simulated User: Powered by GPT-4o, the simulated user holds hidden travel preferences and budget constraints. It reveals them indirectly when the agent asks precise, targeted questions (active elicitation) or after several stalled turns (passive elicitation).
  • Agent Action Space: Agents interact via a unified tool interface supporting three actions:
  • search: Queries a structured database for mock options (flights, hotels, restaurants, car rentals, apartments).
  • action: Communicates directly with the user to ask clarifying questions.
  • answer: Submits a final recommendation ID evaluated against ground-truth preferences and budget utility.
  • Realistic Distractors: Search returns a mix of correct options (satisfying all preferences), wrong options (violating at least one preference), and noise options (irrelevant or corrupted results).
  • Dual Evaluation Settings:
  • Single-choice: The agent must recommend exactly one option per aspect.
  • Multi-choice: The agent can output multiple options, scored by the best choice provided.

Datasets

The authors curated a benchmark consisting of 4,000+ travel planning scenarios (471 test instances, 2,651 reserved for training):

  • Covers 5 core aspects (hotel, restaurant, flight, rental car, apartment).
  • Includes 82 distinct user preferences across 24 categories, paired with 902 carefully validated implicit elicitation statements.
  • Categorized into three difficulty tiers (Easy, Medium, Hard) depending on the number and distribution of preferences per itinerary.

Key Insights and Ideas

  • Tool use is solved; conversation is not: Top models achieved >80% validity on database search syntax, but their valid clarifying action rates hovered below 30–40%.
  • More dialogue turns do not imply better alignment: Giving agents up to 20 or 30 turns did not consistently improve performance. Without explicit dialogue planning, agents frequently devolved into repetitive loops or off-topic chatter.
  • Preference density per aspect is the real bottleneck: Spreading 4 preferences across 4 aspects (e.g., one for flight, one for hotel, etc.) yielded much higher scores than concentrating 4 preferences into a single aspect, showing that local constraint satisfaction quickly overloads agent reasoning.
  • The "Guessing" illusion in multi-choice settings: Agents scored ~40% higher in multi-choice mode compared to single-choice mode. However, their active preference elicitation rates stayed flat or declined—meaning the improvement came from brute-force guessing rather than genuine understanding.
  • Elicitation vs. Integration gap: Claude-4-Sonnet elicited the highest percentage of preferences (over 34% actively/passively), yet GPT-4o achieved higher final task scores. Knowing what the user wants does not automatically translate into choosing the optimal candidate.

Key Results

Under the primary single-choice setting across leading closed- and open-source models:

  • Overall Scores: GPT-4o achieved the highest score (0.329), followed closely by Gemini-2.5-Pro (0.317) and Claude-4-Sonnet (0.307). Open-source models like DeepSeek-V3 (0.210) and Qwen3-32B (0.206) lagged behind.
  • Correct Option Rates: Even top-tier models selected user-aligned options only 31% to 36% of the time (Correct Exist Rate), and the single best cost-effective option only 20% to 26% of the time (Best Exist Rate).
  • Active Elicitation: Top performers proactively uncovered less than 27% of hidden user preferences (GPT-4o: 24.06% active; Gemini-2.5-Pro: 23.85% active; Claude-4-Sonnet: 26.31% active). Smaller models (Llama-3.1-8B, Qwen3-8B) actively elicited under 12%.

Limitations and Future Work

  • Domain Specificity: The environment currently centers on travel planning; extending it to domains like coding assistance, customer service, or healthcare remains open.
  • Simulated User Artifacts: While GPT-4o models human conversational nuances reasonably well, simulated users may exhibit predictability that differs from live human subjects.
  • Balancing Interaction Efficiency and Probing: Models often guess prematurely to minimize turn count. Future work involves using UserBench as a training ground for multi-turn reinforcement learning (RL), designing reward schedules that penalize unprompted guessing while rewarding systematic intent elicitation.

Summary updated

Abstract

Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.
BibTeX
@inproceedings{qian2025userbench,
  author = {Cheng Qian and Zuxin Liu and Akshara Prabhakar and Zhiwei Liu and Jianguo Zhang and Haolin Chen and Heng Ji and Weiran Yao and Shelby Heinecke and Silvio Savarese and Huan Wang},
  title = {{UserBench: An Interactive Gym Environment for User-Centric Agents}},
  booktitle = {SEA @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=iJS7nvlGPd},
}

Loading related content...

Figure