Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents

Adi Krishnamoorthy, X. Y. HanarXiv 2026 · arXiv

TL;DR

For LLM-based shopping agents, "thinking harder" is often a placebo for performance but a stimulant for ego. A new study across ten frontier models reveals that increasing reasoning effort significantly boosts an agent's self-reported confidence without providing a corresponding increase in its ability to choose the correct product.

What Problem Does This Paper Address?

As we move toward "agentic" e-commerce—where AI assistants browse catalogs and make purchases on our behalf—two traits become critical: correctness (did it buy the right thing?) and calibration (does it know when it might be wrong?).

Modern frontier models now offer explicit "reasoning effort" controls (like those found in OpenAI’s o1 or DeepSeek-R1), which allow models to spend more "inference-time compute" (thinking tokens) before answering. While this has been shown to help with structured tasks like math and coding, its utility in "everyday" reasoning—like parsing a messy product listing—remains unclear. The authors investigate whether "thinking harder" actually helps these agents make better choices or if it simply makes them more overconfident.

Methods

The researchers benchmarked ten frontier models (including GPT-6 Astra/Luna, Claude 5.5 Opus, Gemini 4 Argon, and DeepSeek V4.1 Flash) using a standardized decision-making task.

To isolate product judgment from search or navigation errors, they presented each model with a shopping query and a fixed list of ten candidate products (one correct "target" and nine "distractors"). They manipulated the reasoning_effort parameter (low, medium, high) and asked the models to:

  1. Pick the best product.
  2. State their confidence on a scale of 0 to 10.

They then measured success rates, verbalized confidence, and calibration metrics like Expected Calibration Error (ECE)—the gap between stated confidence and actual accuracy.

Datasets

The study utilized three distinct benchmarks to cover various modalities and retailers:

  • WebShop: A multimodal dataset (images + text) based on Amazon listings with a programmatic reward function.
  • Amazon SQID: A multimodal dataset where "correctness" is determined by human relevance judgments.
  • Wayfair WANDS: A text-only dataset focused on furniture and home goods, also using human-labeled ground truth.

Key Insights and Ideas

  • Confidence is Cheap, Correctness is Hard: For most models, increasing reasoning effort led to a statistically significant rise in confidence but a negligible change in accuracy. Essentially, the extra "thinking" tokens often serve to justify a pre-existing (and sometimes incorrect) choice rather than correct it.
  • The "Claude Exception": Calibration is highly model-dependent. While most models were overconfident by 7–33 percentage points, Anthropic’s Claude models (Fable 5.1 and Opus 5.5) were remarkably well-calibrated, sometimes even trending toward slight underconfidence.
  • Shared Failures (Groupthink): Models don't fail randomly. The distribution of errors is bimodal: tasks are either solved by almost everyone or failed by almost everyone. When models fail, they frequently gravitate toward the same incorrect "near-miss" product, suggesting shared inductive biases or training data overlaps.
  • Reasoning vs. Reading: The authors suggest that shopping is a "reading-bound" task rather than a "reasoning-bound" one. If a model misses a secondary constraint (e.g., "alcohol-free") in the initial pass, deliberating longer doesn't necessarily help it find that information if it has already filtered it out.
Figure 1. WebShop at a glance. Per model, the requested reasoning effort is varied from low to high. The graphs display success rate (left), mean stated confidence (middle) and mean reasoning tokens actually used (right) for each reasoning level. All 28 WebShop conditions lie between 74.0% and 88.1% success. We see that Qwen 3.8 Max goes from 394 to 2,033 reasoning tokens per task between low and high effort (488 to 3,926 on SQID), DeepSeek V4.1 Flash from 445 to 634, Gemini 3.1 Pro from 305 to 772 and Gemini 4 Argon from 0 to 774, whereas GPT-6 Astra and GPT-6.1 Sol stay below 100 reasoning tokens per task even at high effort. Similar graphs for SQID and WANDS can be found in Fig. S8. Takeaway: Success rate does not rise significantly from low to high reasoning for most models, while the model output’s stated confidence does.

Key Results

  • Accuracy Plateau: Across 30 model-dataset pairs, only three showed a statistically significant improvement in success rate when moving from low to high effort.
  • Confidence Surge: In contrast, the same shift in effort raised stated confidence significantly for 9 out of 10 models on the WebShop dataset.
  • Calibration Stability: Calibration metrics (ECE, Brier scores) remained largely stable across effort levels. If a model was overconfident at "low" effort, it remained so at "high" effort.
  • Error Correlation: The pairwise error correlation between models was extremely high (0.7–0.8). This means a "second opinion" from a different frontier model is much less useful than one might hope, as they are likely to make the same mistake.
Table 1. Paired low→high change. Teal cells show accuracy changes (mostly insignificant); coral cells show confidence changes (mostly significant).

Limitations and Future Work

The study focused on a fixed-choice set, removing the complexities of search, pagination, and multi-step navigation where reasoning effort might play a larger role. Additionally, the "ground truth" in human-labeled datasets like SQID and WANDS contains inherent noise (subjective relevance), which can depress absolute accuracy scores.

Future work could explore whether these findings hold when agents are allowed to browse autonomously, or if specific training for calibration (rather than just reasoning) can break the "confidence-without-correctness" trap. For now, the takeaway for engineers is clear: don't just turn up the reasoning dial; choose a model that knows when it's guessing.

Summary updated

Abstract

Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
BibTeX
@misc{krishnamoorthy2026thinking,
  author = {Adi Krishnamoorthy and X. Y. Han},
  title = {{Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents}},
  year = {2026},
  eprint = {2611.22033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22033},
}
Figure