CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang ShenarXiv 2026 · arXiv

TL;DR

CARTBench is a new, rigorous benchmark designed to evaluate how LLM shopping assistants handle complex, ambiguous requests that evolve over time. It reveals a significant "understanding-to-execution gap": while top models can follow basic instructions, they struggle to provide high-quality, nuanced advice that accounts for shifting deadlines and unconfirmed user preferences.

What Problem Does This Paper Address?

Most AI shopping benchmarks focus on simple "find me X" tasks. In reality, shopping is messy. A user might need a gift by a specific deadline (which might pass), have conflicting constraints (like fitting everything into one bag), or leave key details unstated (like a preferred feeding plan for a baby).

The authors identify these as CART (Complex and Ambiguous Requests with Timelines) cases. Evaluating LLMs on these scenarios is difficult due to two main hurdles:

  1. Data Scarcity: Real-world commercial dialogue is often confidential and hard to access.
  2. Standardization: It’s hard to create a "one-size-fits-all" checklist for shopping advice because "good advice" for buying a winter coat is fundamentally different from advice for buying a baby kit.

Methods

To solve these issues, the researchers introduced several novel components:

  • Ideal Response Principles: Instead of a generic checklist, they distilled four high-level principles from "excellent" human-written replies: address the current decision, follow active needs while preserving unknowns, ground claims in evidence, and provide useful decision help.
  • Case-Specific Rubrics: Experts used these principles to create tailored rubrics for 169 different cases. These rubrics are divided into three tiers: Core (basic requirements), Excellence (high-quality reasoning and trade-offs), and Safety (avoiding harmful or medical advice).
  • Principle-Guided CART Prompting: To give models a fair shot, they provided them with the "Ideal Response Principles" as part of their instructions, though the specific evaluation rubrics remained hidden.
  • Evidence-Grounded Judging: They used a high-performing LLM (DeepSeek-V4-Pro) to score replies against the rubrics, with a human expert verifying the results to ensure 99.7% agreement.
Figure 1: A CART case: an expired gift deadline (Time- lines), a one-bag limit (Complexity), and an unknown hospital list and feeding plan (Ambiguity). Matched colors link reply excerpts to their checks.

Datasets

The authors introduced CARTBench, which consists of:

  • 169 representative cases curated from real commercial needs across 17 retail domains (e.g., outdoors, personal care, furnishings).
  • 1,603 product records paired with timestamped dialogue histories.
  • 1,328 expert-annotated criteria used for evaluation.

Unlike previous benchmarks like WebShop or EComAgentBench, CARTBench explicitly tracks timelines and rewards models for not guessing when information is missing.

Key Insights and Ideas

  • The "Understanding-to-Execution" Gap: This is the paper's most striking finding. Models are getting very good at "Core" tasks (e.g., not recommending a product that's out of stock), but they fall flat on "Excellence." They can identify the user's goal but struggle to execute a high-quality response that explains trade-offs or prioritizes purchases effectively.
  • Uncertainty Collapse: Models have a tendency to "guess" rather than admit they don't know something. In ambiguous scenarios, they often treat unknown preferences as settled, leading to "lucky guesses" rather than justified, conditional advice.
  • Product Evidence Overreach: A common failure mode (F7) is when models invent features for a product that aren't in the provided data. Another (F8) is "Product Fit Leap," where a model assumes a product is a perfect fit for a user based on a single attribute without sufficient justification.
Figure 3: Case-specific rubrics distinguish stronger and weaker replies to the same warmth and camera-control request through evidence, decision help, and safety checks. Products are reference options, not unique answers. Excerpts show 3 checks.

Key Results

The researchers evaluated 10 models, including flagship APIs like GPT-6.1-Sol and Claude-Opus-5.

  • Core Performance: 3 out of 10 models exceeded 90% on Core requirements. GPT-6.1-Sol led the pack here at 95.9%.
  • Excellence Performance: None of the models exceeded 45%. Claude-Opus-5 performed best in this category at 39.8%, while GPT-6.1-Sol lagged at 29.6%.
  • Safety: Most commercial models performed well, with GPT-6.1-Sol hitting a perfect 100% on safety checks.
  • Scenario Sensitivity: Models struggled most with "Constraints" (interacting requirements). For 7 out of 10 models, this scenario had the lowest "Excellence" rate.

Limitations and Future Work

The authors acknowledge that CARTBench doesn't yet cover highly specialized domains (like historically accurate costumes) or the full post-purchase lifecycle (returns, replacements, and long-term feedback). They suggest that future work should explore how shopping assistants adapt their advice over these longer cycles and how to further bridge the gap between basic instruction following and expert-level decision support.

Summary updated

Abstract

Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at https://github.com/Da1yuqin/CARTBench.
BibTeX
@misc{dai2026cartbench,
  author = {Yuqin Dai and Zhanwei Zhang and Honghui Yang and Zhouheng Yao and Yueqi Zhang and Jingyu He and Chaoyang Shi and Fanding Wei and Boyuan Pan and Kai Xu and Yao Hu and Yongliang Shen},
  title = {{CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines}},
  year = {2026},
  eprint = {2611.05862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05862},
}
Figure