TL;DR
For LLM-based shopping agents, "thinking harder" is often a placebo for performance but a stimulant for ego. A new study across ten frontier models reveals that increasing reasoning effort significantly boosts an agent's self-reported confidence without providing a corresponding increase in its ability to choose the correct product.
What Problem Does This Paper Address?
As we move toward "agentic" e-commerce—where AI assistants browse catalogs and make purchases on our behalf—two traits become critical: correctness (did it buy the right thing?) and calibration (does it know when it might be wrong?).
Modern frontier models now offer explicit "reasoning effort" controls (like those found in OpenAI’s o1 or DeepSeek-R1), which allow models to spend more "inference-time compute" (thinking tokens) before answering. While this has been shown to help with structured tasks like math and coding, its utility in "everyday" reasoning—like parsing a messy product listing—remains unclear. The authors investigate whether "thinking harder" actually helps these agents make better choices or if it simply makes them more overconfident.
Methods
The researchers benchmarked ten frontier models (including GPT-6 Astra/Luna, Claude 5.5 Opus, Gemini 4 Argon, and DeepSeek V4.1 Flash) using a standardized decision-making task.
To isolate product judgment from search or navigation errors, they presented each model with a shopping query and a fixed list of ten candidate products (one correct "target" and nine "distractors"). They manipulated the reasoning_effort parameter (low, medium, high) and asked the models to:
- Pick the best product.
- State their confidence on a scale of 0 to 10.
They then measured success rates, verbalized confidence, and calibration metrics like Expected Calibration Error (ECE)—the gap between stated confidence and actual accuracy.
Datasets
The study utilized three distinct benchmarks to cover various modalities and retailers:
- WebShop: A multimodal dataset (images + text) based on Amazon listings with a programmatic reward function.
- Amazon SQID: A multimodal dataset where "correctness" is determined by human relevance judgments.
- Wayfair WANDS: A text-only dataset focused on furniture and home goods, also using human-labeled ground truth.
Key Insights and Ideas
- Confidence is Cheap, Correctness is Hard: For most models, increasing reasoning effort led to a statistically significant rise in confidence but a negligible change in accuracy. Essentially, the extra "thinking" tokens often serve to justify a pre-existing (and sometimes incorrect) choice rather than correct it.
- The "Claude Exception": Calibration is highly model-dependent. While most models were overconfident by 7–33 percentage points, Anthropic’s Claude models (Fable 5.1 and Opus 5.5) were remarkably well-calibrated, sometimes even trending toward slight underconfidence.
- Shared Failures (Groupthink): Models don't fail randomly. The distribution of errors is bimodal: tasks are either solved by almost everyone or failed by almost everyone. When models fail, they frequently gravitate toward the same incorrect "near-miss" product, suggesting shared inductive biases or training data overlaps.
- Reasoning vs. Reading: The authors suggest that shopping is a "reading-bound" task rather than a "reasoning-bound" one. If a model misses a secondary constraint (e.g., "alcohol-free") in the initial pass, deliberating longer doesn't necessarily help it find that information if it has already filtered it out.
Key Results
- Accuracy Plateau: Across 30 model-dataset pairs, only three showed a statistically significant improvement in success rate when moving from low to high effort.
- Confidence Surge: In contrast, the same shift in effort raised stated confidence significantly for 9 out of 10 models on the WebShop dataset.
- Calibration Stability: Calibration metrics (ECE, Brier scores) remained largely stable across effort levels. If a model was overconfident at "low" effort, it remained so at "high" effort.
- Error Correlation: The pairwise error correlation between models was extremely high (0.7–0.8). This means a "second opinion" from a different frontier model is much less useful than one might hope, as they are likely to make the same mistake.
Limitations and Future Work
The study focused on a fixed-choice set, removing the complexities of search, pagination, and multi-step navigation where reasoning effort might play a larger role. Additionally, the "ground truth" in human-labeled datasets like SQID and WANDS contains inherent noise (subjective relevance), which can depress absolute accuracy scores.
Future work could explore whether these findings hold when agents are allowed to browse autonomously, or if specific training for calibration (rather than just reasoning) can break the "confidence-without-correctness" trap. For now, the takeaway for engineers is clear: don't just turn up the reasoning dial; choose a model that knows when it's guessing.
Summary updated