TL;DR
CARTBench is a new, rigorous benchmark designed to evaluate how LLM shopping assistants handle complex, ambiguous requests that evolve over time. It reveals a significant "understanding-to-execution gap": while top models can follow basic instructions, they struggle to provide high-quality, nuanced advice that accounts for shifting deadlines and unconfirmed user preferences.
What Problem Does This Paper Address?
Most AI shopping benchmarks focus on simple "find me X" tasks. In reality, shopping is messy. A user might need a gift by a specific deadline (which might pass), have conflicting constraints (like fitting everything into one bag), or leave key details unstated (like a preferred feeding plan for a baby).
The authors identify these as CART (Complex and Ambiguous Requests with Timelines) cases. Evaluating LLMs on these scenarios is difficult due to two main hurdles:
- Data Scarcity: Real-world commercial dialogue is often confidential and hard to access.
- Standardization: It’s hard to create a "one-size-fits-all" checklist for shopping advice because "good advice" for buying a winter coat is fundamentally different from advice for buying a baby kit.
Methods
To solve these issues, the researchers introduced several novel components:
- Ideal Response Principles: Instead of a generic checklist, they distilled four high-level principles from "excellent" human-written replies: address the current decision, follow active needs while preserving unknowns, ground claims in evidence, and provide useful decision help.
- Case-Specific Rubrics: Experts used these principles to create tailored rubrics for 169 different cases. These rubrics are divided into three tiers: Core (basic requirements), Excellence (high-quality reasoning and trade-offs), and Safety (avoiding harmful or medical advice).
- Principle-Guided CART Prompting: To give models a fair shot, they provided them with the "Ideal Response Principles" as part of their instructions, though the specific evaluation rubrics remained hidden.
- Evidence-Grounded Judging: They used a high-performing LLM (DeepSeek-V4-Pro) to score replies against the rubrics, with a human expert verifying the results to ensure 99.7% agreement.
Datasets
The authors introduced CARTBench, which consists of:
- 169 representative cases curated from real commercial needs across 17 retail domains (e.g., outdoors, personal care, furnishings).
- 1,603 product records paired with timestamped dialogue histories.
- 1,328 expert-annotated criteria used for evaluation.
Unlike previous benchmarks like WebShop or EComAgentBench, CARTBench explicitly tracks timelines and rewards models for not guessing when information is missing.
Key Insights and Ideas
- The "Understanding-to-Execution" Gap: This is the paper's most striking finding. Models are getting very good at "Core" tasks (e.g., not recommending a product that's out of stock), but they fall flat on "Excellence." They can identify the user's goal but struggle to execute a high-quality response that explains trade-offs or prioritizes purchases effectively.
- Uncertainty Collapse: Models have a tendency to "guess" rather than admit they don't know something. In ambiguous scenarios, they often treat unknown preferences as settled, leading to "lucky guesses" rather than justified, conditional advice.
- Product Evidence Overreach: A common failure mode (F7) is when models invent features for a product that aren't in the provided data. Another (F8) is "Product Fit Leap," where a model assumes a product is a perfect fit for a user based on a single attribute without sufficient justification.
Key Results
The researchers evaluated 10 models, including flagship APIs like GPT-6.1-Sol and Claude-Opus-5.
- Core Performance: 3 out of 10 models exceeded 90% on Core requirements. GPT-6.1-Sol led the pack here at 95.9%.
- Excellence Performance: None of the models exceeded 45%. Claude-Opus-5 performed best in this category at 39.8%, while GPT-6.1-Sol lagged at 29.6%.
- Safety: Most commercial models performed well, with GPT-6.1-Sol hitting a perfect 100% on safety checks.
- Scenario Sensitivity: Models struggled most with "Constraints" (interacting requirements). For 7 out of 10 models, this scenario had the lowest "Excellence" rate.
Limitations and Future Work
The authors acknowledge that CARTBench doesn't yet cover highly specialized domains (like historically accurate costumes) or the full post-purchase lifecycle (returns, replacements, and long-term feedback). They suggest that future work should explore how shopping assistants adapt their advice over these longer cycles and how to further bridge the gap between basic instruction following and expert-level decision support.
Summary updated