$τ$-Rec-Books: A Disclosure-Controlled Benchmark for Requirement Elicitation in Agentic Recommender Systems

Bharath Sivaram Narasimhan, Karthik R. NarasimhanarXiv 2026 · arXiv

TL;DR

$\tau$-Rec-Books is a new benchmark designed to evaluate how effectively AI agents elicit user requirements in conversational recommendation settings. By introducing "disclosure control"—varying whether a user volunteers information, reveals it only when asked, or keeps it hidden until a proposal is rejected—the authors show that even frontier LLMs struggle significantly when preferences aren't stated upfront, with success rates plummeting from 97.5% to just 28.3%.

What Problem Does This Paper Address?

Conversational recommender systems (CRSs) are supposed to feel natural, but they face a major hurdle: requirement elicitation. In the real world, users rarely provide a complete list of constraints (e.g., "I want a sci-fi book from the 90s written by a UK author in Spanish") in their first message. Instead, they reveal needs progressively.

Existing benchmarks often conflate the difficulty of the search task with the difficulty of the dialogue. This paper introduces $\tau$-Rec-Books to isolate the timing of information disclosure. It asks: how well can an agent navigate a conversation when the user is unhelpful, forgetful, or requires the agent to "guess and check" through proposals?

Methods

The researchers developed a framework where each task is held constant in terms of the underlying "ground truth" requirements, but varies across three disclosure conditions:

  1. Volunteer: The user states all requirements in the first turn.
  2. Mixed: The user states one requirement and reveals others only if specifically asked about that field.
  3. Hidden: At least one requirement is never stated. The agent must propose books, receive a rejection, and infer the hidden constraint (e.g., if the user rejects a 2005 book but accepts a 1999 book, the agent might infer a "pre-2000" constraint).

To ensure evaluation is objective and reproducible, the authors used a rule-based simulated user rather than an LLM judge. This user follows a strict protocol for replies (see Figure 3 below). They also tested two different search interfaces: a standard keyword search and a structured search tool with metadata filters.

Table 1. The rule-based user’s replies and the agents’ system prompt (verbatim excerpts).

Datasets

  • Catalog: A newly curated, audited dataset of 20,000 books sourced from Wikidata (CC0 license). It includes metadata for genre, language, country of origin, and publication year.
  • Tasks: 450 tasks across 150 "task families," ranging from simple (2 predicates) to complex (5 predicates).
  • Reference Agents: The authors provide traces for four LLM agents (including DeepSeek and GPT-6.1 Sol) and two scripted baseline agents (Popularity and Eliminating).

Key Insights and Ideas

  • The Disclosure Gradient: The timing of information is a massive bottleneck. Success isn't just about model size; it's about the agent's ability to drive the conversation.
  • The "Asking" Deficit: A primary reason agents fail "Mixed" tasks is that they simply don't ask enough questions. When explicitly prompted to ask before recommending, success rates for all agents jumped to over 93%.
  • Untested Commitments: In "Hidden" tasks, the most common failure mode (78% of cases) was the agent recommending a book it had never previously proposed to the user for a verdict. Agents tend to "hallucinate" that they have satisfied hidden constraints without verifying them.
  • Structured Search Wins: Providing agents with structured filters (genre, year range, etc.) improved success by 7.1 points on average compared to keyword-only search, as it allowed for more precise hypothesis testing.

Key Results

The study found a sharp decline in performance as information became harder to access. As shown in Figure 2(a), the "disclosure gap" is consistent across all tested models.

  • Satisfiable Task Success (pass^1):
    • Volunteer: 0.975
    • Mixed: 0.756
    • Hidden: 0.283
  • Model Variance: Interestingly, GPT-6.1 Sol (a frontier model) had the lowest success on Mixed tasks because it was the most "reluctant" to ask clarifying questions, though it caught up when the system prompt was modified to enforce an "ask-and-verify" protocol.
  • Scripted Logic vs. LLMs: A scripted "Eliminating" agent—which uses formal logic to rule out possibilities based on rejections—vastly outperformed all LLMs on hidden tasks (0.956 success vs. the LLM average of 0.283).
Figure 2. Success rates by condition and prompt type.

Limitations and Future Work

  • Hard Filters Only: The benchmark currently focuses on "hard" metadata constraints (e.g., "published after 2000"). It does not yet account for "soft" preferences like writing style, mood, or "books similar to X."
  • Catalog Bias: The Wikidata source results in a catalog skewed toward Anglo-American works and Science Fiction.
  • User Realism: While the rule-based user ensures a "clean" experiment, real humans are more nuanced. Future work aims to validate these findings against human-in-the-loop studies and incorporate "critiquing" (where users change their minds after seeing results).

Summary updated

Abstract

While conversational recommendation systems provide a more natural user interface, they face the challenge of progressively discovering a user's needs before they can recommend an item that satisfies them. To evaluate the ability of AI agents at this elicitation process, we introduce $τ$-Rec-Books, a benchmark that holds each user request fixed while varying only when the user reveals their requirements: all at the start (volunteer), one at the start and the rest when asked (mixed), or some never (hidden). A hidden requirement can be found only by proposing books and inferring it from the user's accept or reject verdicts. We generate 450 tasks over a catalog of 20,000 books from Wikidata, develop a rule-based simulated user to answer the agent, and score recommendations against catalog metadata without an LLM judge. Across four agents and two search interfaces, success rates fall for every agent and interface, on average from 0.975 (volunteer) to 0.756 (mixed) and 0.283 (hidden). Mixed tasks fail mainly because agents do not ask, and when told to ask, every agent reaches at least 0.931. On hidden tasks, most failures come from agents recommending a book they never proposed to the user. When told to propose first, agents only match a simple baseline that proposes popular books, and stay well below one that uses rejection to rule out possible requirements. We release the catalog pipeline, task generator, simulated user, validators, reference agents, and all agent traces at https://github.com/nbharaths/tau-rec/tree/main/data/tracks/books-wikidata-v2.
BibTeX
@misc{narasimhan2026rec,
  author = {Bharath Sivaram Narasimhan and Karthik R. Narasimhan},
  title = {{$τ$-Rec-Books: A Disclosure-Controlled Benchmark for Requirement Elicitation in Agentic Recommender Systems}},
  year = {2026},
  eprint = {2611.04459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04459},
}
Figure