Deploying Unified BERT Moderation Model for E-Commerce Reviews

Ravindra Nayak, Nikesh GareraEMNLP 2022

TL;DR

This paper introduces a unified BERT-based model for moderating e-commerce reviews, addressing the challenge of high volume and diverse content. By incorporating product vertical embeddings and leveraging pre-training on monolingual data, their "In-House-BERT-vertical" model significantly reduces the need for manual moderation, achieving high accuracy while serving live traffic.

What Problem Does This Paper Address?

The explosion of user-generated content, particularly product reviews on e-commerce platforms, presents a significant moderation challenge. While reviews build trust, their sheer volume makes manual moderation infeasible due to inconsistency, real-time limitations, lack of domain knowledge, and multilingual complexities. Existing automated systems often struggle with nuanced content, leading to the need for more robust and efficient moderation solutions that can handle various rejection reasons beyond just profanity and hate speech. This paper tackles the problem of effectively moderating a wide range of e-commerce review content, including issues like poor formatting, irrelevance, and mismatches between ratings and sentiment, all while aiming for high accuracy and minimal manual intervention.

Methods

The core of the proposed solution is a unified BERT model, "In-House-BERT-vertical." Key methodological contributions include:

  • Unified BERT Architecture: Instead of separate models for different rejection reasons, they developed a single BERT encoder that outputs review and vertical embeddings. These are then fed into multiple classification heads, each responsible for a specific moderation task (e.g., predicting rating, detecting irrelevance).
  • Product Vertical Embeddings: To improve the detection of irrelevant reviews, they incorporate product vertical (category) information as an embedding. This helps the model understand if a review is genuinely about the product it's associated with.
  • Pre-training on Monolingual Data: To address the domain gap and scarcity of labeled data, they pre-trained a BERT model ("In-House-BERT") on a large corpus of monolingual e-commerce reviews and product descriptions. This pre-training helps the model learn domain-specific language and representations.
  • Data Augmentation: To enhance robustness against adversarial attacks and noisy data, they applied various augmentation techniques to the training data, such as character replacement, vowel dropping, and random case changes, particularly for profanity detection.
  • Hybrid Approach (Deterministic + Model-based): The system combines rule-based methods (regex parsers, list matching for profanity) with the BERT model. This hybrid approach allows for deterministic rejection of clearly problematic content (e.g., email addresses, phone numbers) while using the BERT model for more complex classification tasks.
  • Thresholding for Inference: For inference, they employ thresholds for different rejection reasons. This allows for stricter rejection of certain categories where recall is paramount, and reviews falling below these confidence thresholds are sent for manual review.

Datasets

  • In-house Monolingual Data: A large corpus of product descriptions and reviews (approximately 1 billion tokens) was used for pre-training the "In-House-BERT" model.
  • Manually Labeled Corpus: A custom dataset was created by manually labeling reviews according to their moderation guidelines. This dataset was split into training, validation, and test sets.
  • Smaller Dataset (16k examples): A smaller dataset was created to specifically evaluate the benefits of pre-training when labeled data is scarce.

Key Insights and Ideas

  • Pre-training is Crucial for Scarce Labeled Data: The paper strongly demonstrates that pre-training BERT on domain-specific monolingual data significantly boosts performance (4.78% F1 increase) when labeled data is limited. This is a key takeaway for practical deployment in real-world scenarios where extensive labeled datasets are costly to obtain.
  • Product Context Matters for Relevance: Incorporating product vertical embeddings proved effective in improving the detection of irrelevant reviews, highlighting the importance of contextual information beyond just the review text itself.
  • Freezing Layers Reduces Training Time: Freezing the initial layers of the BERT model during fine-tuning significantly reduces training time (by up to 40%) without a substantial drop in performance, making the training process more efficient.
  • Unified Model Simplifies Deployment: A single, unified model architecture for multiple moderation tasks simplifies deployment and maintenance compared to managing numerous specialized models.
  • Hybrid Approach Balances Automation and Accuracy: Combining deterministic rules with a sophisticated BERT model offers a practical balance, ensuring clear-cut cases are handled efficiently while leveraging deep learning for more complex content.

Key Results

  • Overall Performance: The "In-House-BERT-vertical" model achieved an F1 score of 88.45%, outperforming publicly available BERT-based models by 2.57%.
  • Reduced Manual Moderation: The deployed system significantly reduced the volume of reviews sent for manual moderation from 23% to 5.89%.
  • Performance on Smaller Datasets: On a smaller dataset, pre-training led to a notable 4.78% increase in F1 score.
  • Latency and Throughput: The system demonstrated a P95 latency of 120 ms on 2-core CPUs with 2 GB RAM, handling up to 10 queries per second in production.
  • Binary Classification: When evaluated as a binary classification problem (approved vs. reject), the best model achieved an F1 score of 93.02%.

Limitations and Future Work

  • Multilingual Support: The current system primarily focuses on English text. Expanding to support multilingual, multi-script, and code-mixed moderation is a key future direction.
  • Explainability: The authors aim to improve the explainability of their model to better convey rejection reasons to users.
  • Adversarial Attacks and Noisy Labels: Further work is planned to make the model more robust against adversarial attacks and noisy label tagging.
  • Imbalanced Datasets: Addressing imbalanced datasets for certain rejection reasons will be a focus.
  • Additional Rejection Reasons: Future work includes incorporating detection for sarcasm and opinion spam.

Summary updated

BibTeX
@inproceedings{nayak2022deploying,
  author = {Ravindra Nayak and Nikesh Garera},
  title = {{Deploying Unified BERT Moderation Model for E-Commerce Reviews}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure