TL;DR
While generative AI models for e-commerce question answering achieve high scores on standard linguistic metrics, they often fail at the actual task. This paper reveals that state-of-the-art models frequently ignore the product reviews they are supposed to summarize, struggle with basic numerical facts, and are heavily biased toward uninformative "I don't know" responses.
What Problem Does This Paper Address?
In e-commerce, Product Question Answering (PQA) platforms allow customers to ask questions that other users answer based on their experiences. Because manually answering every query is impossible, researchers have turned to generative models—AI that reads product reviews and writes a custom answer.
The problem is that current evaluation relies almost entirely on ROUGE scores, which measure how many words a generated answer shares with a human-written reference. The authors argue that ROUGE is a "shallow" metric: a model can get a high score by using the right words while providing factually incorrect or completely useless information. This paper investigates whether these models are actually learning to answer questions or just performing clever pattern matching.
Methods
The authors performed a rigorous "stress test" on two state-of-the-art generative PQA models:
- OAAG (Opinion-aware Answer Generation): A Bi-LSTM model using attention mechanisms and a pointer-generator network to copy words from reviews.
- CHIME (Cross-passage Hierarchical Memory Network): A more modern transformer-based model (using XLNet) with an auxiliary memory module to aggregate information across multiple reviews.
Instead of just reporting scores, the researchers designed experiments to test:
- Review Utilization: They replaced relevant reviews with random ones to see if the models actually "read" the input.
- Numerical Reasoning: They isolated questions involving dimensions, weights, or counts.
- Dataset Bias: They analyzed how frequently the models defaulted to uninformative phrases found in the training data.
Datasets
The study utilized two massive, standard benchmarks:
- Amazon Question Answering Dataset: Contains ~1.4 million user questions and answers across various categories.
- Amazon Product Review Dataset: Provides the context (reviews) used by the models to generate answers.
- Focus Areas: The authors specifically evaluated the "Home & Kitchen" and "Sports & Outdoors" categories.
Key Insights and Ideas
- The "Random Review" Paradox: Surprisingly, when the authors gave the models random reviews from completely different products, the performance (ROUGE) barely dropped. This suggests the models are often ignoring the provided context and relying on internal biases or the question text alone.
- The "I Don't Know" (IDK) Trap: Real-world training data is messy; many humans answer questions with "I don't know." The models over-learn this. In the "Home" category, over 35% of generated responses started with uninformative phrases like "I don't think so," even when the answer was likely available in the reviews.
- Numerical Illiteracy: The models are "abysmal" at numbers. They might correctly identify that a question is about "inches," but they frequently hallucinate the actual value. For example, if a reference says "3 inches," the model might confidently output "1.5 inches."
- ROUGE is Blind to Logic: The authors found cases where a model's answer was the exact opposite of the truth (e.g., saying "it is not wood" when the reference says "it is wood") yet still received a high ROUGE score because of the word overlap.
Key Results
- Correctness Gap: Upon manual inspection of 100 high-scoring samples, the authors found that 54% (OAAG) and 56% (CHIME) were actually incorrect, despite their high ROUGE scores.
- Numerical Failure: In a manual check of 400 numerical answers for the OAAG model, only two were correct. For CHIME, zero out of 100 random numerical samples were correct.
- Bias Impact: By simply removing "IDK" style answers from the training set, the authors were able to significantly improve the ROUGE scores for the "Home" category, proving that the models were previously wasting capacity on uninformative patterns.
Limitations and Future Work
The paper identifies critical flaws but does not propose a new architecture to fix them. Instead, it serves as a "call to arms" for the NLP community.
The authors suggest that future PQA research must:
- Move beyond ROUGE: Develop metrics that prioritize factual correctness and information utility.
- Improve Grounding: Ensure models are strictly tied to the evidence provided in reviews.
- Handle Numbers: Incorporate specific modules for numerical reasoning.
- Data Cleaning: Better filter training data to prevent models from learning uninformative "filler" responses.
Summary updated