Introduction
Building effective shopping assistants requires a deep understanding of user intent. While general-purpose AI models excel at broad question answering, they often struggle in specialized domains like e-commerce. This is largely due to "one-size-fits-all" taxonomies that fail to capture the pragmatic nature of shopping queries. For example, a general model might classify a request for a product recommendation as a "Debate" because of its subjective nature, leading to poor downstream performance.
Researchers from UC Santa Cruz and Amazon addressed this by developing EQA (E-commerce Question Answering), a bespoke taxonomy designed specifically for the shopping context. Their work demonstrates that domain-specific intent classification significantly improves human annotator agreement and provides a more robust foundation for training machine learning models.
Methodology
The researchers adopted a "bespoke strategy," constructing a taxonomy from the ground up rather than adapting existing open-domain frameworks like NFQA (Non-Factoid Question Answering).
Taxonomy Design
The EQA taxonomy identifies six core categories: Instruction, Opinion, Description, Comparison, Recommendation, and Factoid. By removing irrelevant categories such as "Debate" and introducing "Recommendation," the system aligns more closely with the actual information needs of shoppers.
Implementation and Training
To build a functional classifier, the team utilized Flan-T5-Large, an encoder-decoder model. They employed instruction fine-tuning, prompting the model to output specific category labels based on the input question. To ensure reliability, they implemented a confidence scoring mechanism based on token transition probabilities, allowing the system to flag uncertain predictions (below a 0.6 threshold).
The models were trained and evaluated using the AmazonQA dataset, which contains over 92,000 question-answer pairs from product reviews, and a curated human evaluation set of 350 questions to validate taxonomy clarity.
Key Results
The shift from a general taxonomy to a bespoke one yielded dramatic improvements in both data quality and model performance.
Improved Human Agreement
The most significant finding was the jump in human annotator agreement. When using the general NFQA taxonomy, humans agreed on labels only 49.13% of the time. With the EQA taxonomy, this agreement rate soared to 76.88%. This suggests that EQA is far more intuitive and better suited for describing e-commerce data.
Realistic Intent Distribution
Under the old NFQA framework, over 55% of questions were lumped into the "Debate" category. EQA redistributed these into more actionable categories: 52% Factoid, 17% Opinion, and 15% Description. Qualitatively, EQA proved much better at handling shorthand queries (e.g., "10' is L, H or W?") that general models often dismissed as "Not-a-question."
Conclusion
The EQA taxonomy demonstrates that context is paramount in NLP. By focusing on the pragmatic needs of shoppers—facilitating a purchase rather than engaging in philosophical discussion—the researchers created a more reliable system for intent understanding. While the direct impact on downstream tasks like search ranking is still being explored, this "bespoke" methodology provides a blueprint for other specialized fields like healthcare or law, where general-purpose AI often falls short.
Summary updated