Leveraging Seq2seq Language Generation for Multi-level Product Issue Identification

Yang Liu, Varnith Chordia, Hua Li, Siavash Fazeli Dehkordy, Yifei Sun, Vincent Gao, Na Zhang2022 · DOI

Abstract

In a leading e-commerce business, we receive hundreds of millions of customer feedback from different text communication channels such as product reviews. The feedback can contain rich information regarding customers' dissatisfaction in the quality of goods and services. To harness such information to better serve customers, in this paper, we created a machine learning approach to automatically identify product issues and uncover root causes from the customer feedback text. We identify issues at two levels: coarse grained (L-Coarse) and fine grained (L-Granular). We formulate this multi-level product issue identification problem as a seq2seq language generation problem. Specifically, we utilize transformer-based seq2seq models due to their versatility and strong transfer-learning capability. We demonstrate that our approach is label efficient and outperforms the traditional approach such as multi-class multi-label classification formulation. Based on human evaluation, our fine-tuned model achieves 82.1% and 95.4% human-level performance for L-Coarse and L-Granular issue identification, respectively. Furthermore, our experiments illustrate that the model can generalize to identify unseen L-Granular issues.

Introduction

Identifying product issues from massive volumes of customer feedback is critical for improving e-commerce quality and service. However, traditional classification methods face significant hurdles. First, the set of potential issues is dynamic; new problems emerge constantly, making fixed-label classifiers brittle. Second, stakeholders require information at different granularities: high-level trends for business strategy and specific root causes for engineering fixes. Finally, customer feedback is often noisy, containing irrelevant text or multiple overlapping issues.

To address these challenges, we propose a shift from traditional classification to a sequence-to-sequence (seq2seq) generation approach. By treating issues as natural language strings rather than category IDs, a single model can identify both broad categories and specific, free-form descriptions, even for issues not seen during training.

Methodology

We formulate issue identification as a text-to-text task where the model generates literal strings describing product defects. We evaluated several transformer-based architectures, including T5, BART, and Pegasus, with T5 demonstrating the most consistent performance across granularity levels.

The proposed multi-level issue identification framework using task-specific prefixes and a transformer-based seq2seq model.

To enable a single model to handle multiple tasks, we utilize task-specific prefixes such as "identify high-level issues:" or "identify fine-grain issues:". This guides the model to the appropriate level of abstraction. Our training process also incorporates auxiliary tasks—such as text summarization and token span infilling—to improve the model's understanding of product-related nuances. Additionally, we employ sentence and issue shuffling to ensure the model learns content rather than memorizing specific sequences.

Results and Discussion

Our generative approach significantly outperforms traditional multi-label classifiers. For instance, a BERT-based multi-label classifier (MCML-BERT) showed a 24.1% lower ROUGE-1 score on coarse issues compared to our seq2seq model.

A key finding is the model's exceptional performance on granular issues. As shown in the table below, the model achieves 95.4% of human-level performance for fine-grained identification.

Model performance relative to human benchmarks for coarse and granular issue identification.

The higher performance on L-Granular issues (95.4%) compared to L-Coarse issues (82.1%) suggests that models excel at "literal" extraction from source text, whereas abstract categorization remains more challenging. Notably, the model demonstrates strong zero-shot generalization; it can correctly identify "unseen" granular issues approximately 70% of the time by describing the feedback content directly.

To better evaluate these generative outputs, we moved beyond standard ROUGE metrics to include SimCSE Precision and Recall. These metrics use sentence embeddings to assess semantic similarity, ensuring that the model is rewarded for correct identification even if its wording differs from the ground truth.

Conclusion

By reimagining product issue identification as a language generation problem, we have developed a flexible, label-efficient system capable of near-human performance on granular root-cause identification. While a performance gap remains in high-level abstract categorization, the ability of generative models to handle dynamic, unseen issues represents a significant advancement over traditional classification methods. Future work will focus on closing the abstraction gap and further refining zero-shot capabilities.

Summary updated

Abstract

In a leading e-commerce business, we receive hundreds of millions of customer feedback from different text communication channels such as product reviews. The feedback can contain rich information regarding customers' dissatisfaction in the quality of goods and services. To harness such information to better serve customers, in this paper, we created a machine learning approach to automatically identify product issues and uncover root causes from the customer feedback text. We identify issues at two levels: coarse grained (L-Coarse) and fine grained (L-Granular). We formulate this multi-level product issue identification problem as a seq2seq language generation problem. Specifically, we utilize transformerbased seq2seq models due to their versatility and strong transfer-learning capability. We demonstrate that our approach is label efficient and outperforms the traditional approach such as multi-class multi-label classification formulation. Based on human evaluation, our fine-tuned model achieves 82.1% and 95.4% human-level performance for L-Coarse and L-Granular issue identification, respectively. Furthermore, our experiments illustrate that the model can generalize to identify unseen L-Granular issues.
BibTeX
@misc{liu2022leveraging,
  author = {Yang Liu and Varnith Chordia and Hua Li and Siavash Fazeli Dehkordy and Yifei Sun and Vincent Gao and Na Zhang},
  title = {{Leveraging Seq2seq Language Generation for Multi-level Product Issue Identification}},
  year = {2022},
  doi = {10.18653/v1/2022.ecnlp-1.3},
}
Figure