DEFTri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commerce

Ipsita MohantyACL 2022

TL;DR

This paper introduces DEFTri, a novel framework for automated product defect triage in e-commerce. By fusing defect text with team label information and leveraging a fine-tuned BERT model, DEFTri significantly improves the accuracy of assigning defects to the correct teams, addressing a critical bottleneck in large-scale software development.

What Problem Does This Paper Address?

In large e-commerce companies, a massive volume of product defects are generated daily. Efficiently routing these defects to the specialized teams responsible for their resolution is crucial for timely software releases. The current manual triage process, often handled by human agents, is prone to bottlenecks and inefficiencies. Misassigning defects can lead to wasted time diagnosing issues outside a team's expertise, ultimately impacting defect resolution throughput. While automated defect triage has been explored, the application of state-of-the-art pre-trained language models, particularly in the context of e-commerce defect data, remains an under-explored area. This paper aims to bridge this gap by proposing an automated solution that leverages contextual representations of defect texts and their associated labels.

Methods

The core of DEFTri lies in its novel approach to learning contextual representations for multi-label defect classification. The authors propose two main implementations of a label-fused BERT model:

  • Label Fused Model with [SEP]: This approach treats the problem as a sentence-pair task for BERT. Team labels are concatenated as "Sentence A," and the defect title and description form "Sentence B," separated by a [SEP] token. This allows BERT to learn the relationship between the defect text and each potential team label.
  • Label Fused Model without [SEP]: In this variation, the team labels and defect text are concatenated into a single sequence ("Sentence A") without the [SEP] token. This aims to create a more integrated representation of the defect and its associated labels.

Both models are fine-tuned on a BERT base uncased pre-trained transformer. The classification head uses either a linear layer or a BiLSTM layer. For training, they employ BCEWithLogitsLoss and the AdamOptimizer.

Datasets

The paper introduces a proprietary, in-house dataset of actual product defect reviews generated by beta testers for a major software release. This dataset consists of defect titles and descriptions.

  • Training Set: 3485 samples
  • Test Set: 85 samples
  • Team Labels: 15 unique team labels

Crucially, this dataset is characterized as low-resource, with only 4-5 human-expert annotated defects per team label. To address this, the authors employ two key data generation strategies:

  1. Weak Supervision (Snorkel): They utilize the Snorkel framework with 25 labeling functions (LFs) to generate weak labels for the training data from unlabeled defect reviews.
  2. Adversarial Learning (Textattack): To increase the training sample size and robustness, they generate synthetic data using the Textattack framework. This involves augmenting text by replacing words with neighbors in an embedding space, ensuring a cosine similarity of at least 0.8, and altering approximately 10% of original text words.

The paper also notes that prior research often uses open-source datasets like Eclipse and Mozilla, but these datasets focus on technical errors from system failures and don't fully capture the natural language, technical, and domain-specific text found in e-commerce defect reviews.

Key Insights and Ideas

  • Label Fusion is Key: The most significant insight is that fusing team label information with defect text embeddings dramatically improves defect triage performance compared to using defect text alone. This suggests that the semantic meaning of team labels is highly informative for classification.
  • [SEP] Token's Role: The experimental results indicate that the "Label Fused Model without [SEP]" performed better than the one with [SEP]. The authors hypothesize this might be due to the "unnatural formation" of Sentence A when a long list of team labels is concatenated with the [SEP] token.
  • Adversarial Data Boosts Performance: Synthetically generated adversarial data led to an average accuracy improvement of 2.69% across models, highlighting the benefit of increased data diversity and robustness training.
  • Low-Resource Data Generation is Viable: The paper demonstrates a practical approach to creating a usable dataset for a low-resource, multi-label classification task by combining weak supervision and adversarial learning.

Key Results

The DEFTri framework, particularly the BERT+LabelFuse w/o [SEP]+BiLSTM model, achieved the highest performance, reaching a Macro-F1 score of 0.8245 and an Accuracy of 0.8241. This significantly outperforms the baseline BERT+Linear model (Macro-F1: 0.8123, Accuracy: 0.8134). The label-fused models consistently outperformed the base BERT model. The addition of adversarial data resulted in an average accuracy improvement of 2.69%.

Limitations and Future Work

Limitations:

  • Sensitivity to Adversarial Data Parameters: The authors note that performance was sensitive to the choice of text corpus sequence length and perturbation percentage during data augmentation. Higher perturbations with lower sequence lengths negatively impacted performance.
  • Error Propagation from Pseudo-Labels: While weak supervision solves low-labeled data issues, it can suffer from error propagation from pseudo-labels.

Future Work:

  • Contrastive Learning: The authors suggest exploring contrastive learning for fine-tuning using weak supervision, citing recent research showing promise in this area.
  • Improved Adversarial Approaches: Further research into contrastive self-regularized self-training and GAN-BERT in adversarial settings is proposed to enhance performance.

Summary updated

BibTeX
@inproceedings{mohanty2022deftri,
  author = {Ipsita Mohanty},
  title = {{DEFTri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commerce}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure