Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures

Assaf Hallak, Yftah Ziser, Gal ChechikarXiv 2026 · arXiv

TL;DR

Evaluating LLMs as product advisors is difficult because "good advice" isn't a single ground-truth string. This paper introduces a diagnostic framework with 19 specific failure modes—ranging from factual errors to poor trade-off logic—and demonstrates that even frontier models often prefer community-recommended products over their own when presented side-by-side.

What Problem Does This Paper Address?

When you ask an LLM for a car recommendation, how do we know if the answer is actually good? Traditional NLP metrics often rely on "reference matching" (did the model name the same product as a human expert?). However, in product advice, multiple products might be suitable, and human "references" from online forums can be biased or flat-out wrong.

The authors identify a major gap: we lack a systematic way to pinpoint why a model’s advice fails. Is it because the model doesn't know the product exists (coverage)? Is it because it hallucinated a feature (fact error)? Or did it simply fail to weigh the user's preferences correctly (ranking)? Without a granular breakdown, improving these systems is guesswork.

Methods

The researchers propose a structured reasoning process for product advice, decomposing it into six stages:

  1. Evidence ($F, B$): Extracting stated facts and forming background beliefs.
  2. Criteria ($C, P$): Defining hard constraints (e.g., "must seat six") and soft preferences.
  3. Options ($I, I_{filtered}$): Generating a candidate pool and filtering out those that violate constraints.
  4. Choice ($R, p$): Ranking the remaining options and making a final pick.

Based on this, they define 19 failure modes (e.g., "A relevant fact is omitted," "An allowed class is prematurely excluded," "The final pick contradicts the explanation").

To test these, they used:

  • Source-blind comparisons: Models were shown their own recommendation alongside a community-suggested one and asked to pick the best without knowing which was which.
  • Fact anchoring: Providing the model with a pre-extracted list of facts from the user's request to see if "fixing" the evidence stage improved the final choice.
  • Candidate Probes: Forcing models to generate long lists of products to see if the "correct" reference was even in their internal search space.

Datasets

The study utilizes several diverse datasets to test different domains:

  • Reddit Corpus: ~1,800 requests for cars, laptops, and headphones from 2019.
  • Stack Exchange: ~3,700 queries from the "Software Recommendations" and "Hardware Recommendations" communities.
  • ESCI (Shopping Queries Dataset): 500 US-based Amazon search queries used to test selection against externally assigned relevance labels ("Exact" vs. "Irrelevant").

Key Insights and Ideas

  • The "Self-Doubt" Effect: In source-blind trials, models like GPT-5.4 and Llama-70B preferred a community-listed alternative over their own initial recommendation in roughly 6.4–6.5% of cases. This suggests a mismatch between a model's generative capabilities and its evaluative judgment.
  • Naming $\neq$ Selecting: A model might be able to list a product in a "top 40" list (coverage) but fail to select it as the "best" option (selection). For headphones, coverage was only ~57%, meaning the model often didn't even consider the community's favorite products.
  • Fact Anchoring is Not a Silver Bullet: Interestingly, pre-filling the model with a list of facts from the request did not significantly improve agreement with community references. This implies that failures often happen in the filtering and ranking stages, not just in initial fact extraction.
  • Domain Sensitivity: LLMs are much better at "covering" the laptop market than the headphone market, likely due to the higher density of structured data and reviews available for laptops in training sets.

Key Results

  • Constraint Satisfaction: Mistral-14B showed the highest rate of confirmed conflicts (3.8% of car recommendations violated explicit user constraints like drivetrain or transmission). Frontier models like GPT-5 and Claude Opus had zero confirmed conflicts in the specific car audit.
  • Selection Accuracy: On the ESCI (Amazon) dataset, Claude and GPT-5 selected the "Exact" labeled product ~70-74% of the time, significantly outperforming Llama-8B (~58.6%).
  • Reference Agreement: Overall agreement with community-listed products was surprisingly low across all models (14.7% to 27.4%). This highlights that "matching the community" is a poor proxy for quality, as models often make different (and sometimes more defensible) trade-offs.

Limitations and Future Work

  • Temporal Gap: The Reddit data is from 2019, while the models are much newer. While the authors tried to account for this, the "historical availability" of products remains a confounding factor.
  • Prototype Verifiers: The 19-mode diagnostic tool is currently a set of automated heuristics and generative judges. The authors note that these verifiers themselves need further validation against human experts to ensure they aren't flagging "harmless paraphrases" as errors.
  • Future Direction: The authors suggest that the next step is targeted interventions—such as using catalog-backed generation to fix "missing candidate" errors or improving the ranking logic when a model rejects a perfectly suitable product.

For more details, you can find the full paper under the title "Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures" (arXiv:2611.20734v1).

Summary updated

Abstract

Evaluating product advice from a language model requires assessing whether the recommendation meets the user's needs and identifying where the model falls short when it does not. However, matching a product mentioned in an online reply cannot answer either question on its own: several products may be suitable, and the recorded answer may itself be poor advice. When shown a community-listed alternative alongside their own sampled car recommendation, six models prefer the alternative in 1.9 to 6.5 percent of eligible disagreements, consistently across both source-blind presentation orders. We propose a diagnostic framework that separates the user's stated needs, the model's added assumptions, candidate generation, filtering, and final choice, with 19 possible failure modes and the evidence needed to assess them. We apply this framework to product-advice data and to a separate product-selection task with externally assigned relevance labels. The experiments distinguish naming a community-listed product from selecting it and show how answer extraction and candidate-list construction affect the outcome.
BibTeX
@misc{hallak2026can,
  author = {Assaf Hallak and Yftah Ziser and Gal Chechik},
  title = {{Can LLMs Give Good Product Advice? A Framework for Diagnosing Failures}},
  year = {2026},
  eprint = {2611.20734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20734},
}
Figure