Snippext: Semi-supervised Opinion Mining with Augmented Data

Zhengjie Miao, Yuliang Li, Xiaolan Wang, Wang-Chiew TanarXiv 2020 · DOI · arXiv

TL;DR

Snippext is a semi-supervised opinion mining framework that drastically reduces the amount of labeled data needed to fine-tune state-of-the-art language models. By combining a novel text data augmentation technique (MixDA) with a semi-supervised learning framework (MixMatchNL), it matches or exceeds previous state-of-the-art results using half—or in some cases, a third—of the training data.

What Problem Does This Paper Address?

Opinion mining—the task of extracting aspects, opinions, and sentiments from user reviews (e.g., identifying that "the elevator was a bit slow" represents a negative sentiment toward the hotel's infrastructure)—is vital for e-commerce and online services.

Figure 3: The tagging model identifies all aspect (AS) and opinion (OP) spans. Among all candidate pairs of AS-OP spans, the pairing model identifies the correct associations, e.g., (“very nice”, “Everybody”) is correct but not (“very nice”, “food”). Finally, there are two classifiers decide: (1) which attribute that an extracted pair should be assigned to and (2) the sentiment (positive, negative, or neutral) of the opinion.

While pre-trained language models like BERT have achieved remarkable success when fine-tuned for these tasks, they remain notoriously data-hungry. Collecting high-quality, domain-specific labeled training data requires significant human effort, making it both expensive and time-consuming. The authors address a critical question: Can we significantly reduce the reliance on labeled data while maintaining or improving state-of-the-art performance?

Methods

Snippext introduces a powerful two-prong approach to maximize data efficiency by integrating data augmentation with semi-supervised learning:

Figure 2: Overall Architecture of Snippext. MixDA augments and interpolate the labeled training sequences. MixMatchNL further combines labeled and unlabeled data into supervisory signal for fine-tuning the pre-trained Language Model (LM).
  1. MixDA (Data Augmentation): Naive text augmentation (like randomly deleting or swapping words) often destroys syntactic structure or accidentally flips the sentiment of a sentence. Snippext solves this by protecting "target spans" (the actual aspects/opinions) and only altering non-target tokens. It also introduces a new Span-level Replacement (SPR) operator. Crucially, instead of feeding purely augmented text to the model, it uses MixUpNL—a technique that performs convex interpolation between the original and augmented sequences within the BERT embedding space rather than the raw text space. This smooths out the noise introduced by augmentation.
Figure 5: MixDA augments by interpolating over the BERT encoding.
  1. MixMatchNL (Semi-Supervised Learning): Adapted from the popular MixMatch framework in computer vision, this algorithm leverages massive amounts of unlabeled data. For every unlabeled review, the model "guesses" a soft label by averaging and sharpening predictions across multiple augmented views of that review. It then mixes these guessed-label examples with known labeled data using the MixUpNL embedding interpolation technique to regularize the model's decision boundaries.

Datasets

The authors evaluated Snippext on two distinct fronts:

  • ABSA Benchmarks: Four benchmark datasets from SemEval (2014 and 2016) covering two domains: Restaurants and Laptops. The tasks evaluated were Aspect Extraction (AE) and Aspect Sentiment Classification (ASC).
  • Real-World Corpus: A large proprietary dataset consisting of 842,260 hotel reviews from 494 San Francisco hotels, supplemented by 50,000 unlabeled sentences for semi-supervised training.

Key Insights and Ideas

  • Interpolation in the Hidden Space: Because text is discrete, you cannot linearly blend the sentence "The food was good" with "The drinks were great." By performing convex interpolation on the continuous hidden representations output by BERT, Snippext successfully brings a powerful computer vision regularization technique to NLP.
  • Syntactic Safety via Targeted Operators: The paper demonstrates that data augmentation for sequence labeling requires surgical precision. Leaving the core aspect/opinion tokens untouched while varying the surrounding context prevents the label corruption that plagues simpler text augmentation strategies.
  • The Power of "Soft" Labels: By averaging and sharpening predictions on unlabeled data, the model creates highly reliable training signals from unannotated text, effectively mapping out the broader data distribution without human intervention.

Key Results

  • Sashing Data Requirements: In low-resource settings (using only 1,000 training examples), MixMatchNL matched or outperformed the previous state-of-the-art (BERT-PT) which was trained on the entire dataset.
Performance curves of Snippext variants across different training set sizes.
  • New State-of-the-Art: When allowed to use the full training datasets, Snippext established new SOTA benchmarks across all four SemEval tasks, outperforming standard BERT fine-tuning by up to 3.55% on restaurant sentiment classification.
  • Real-World Impact: Deployed on the hotel review corpus, Snippext extracted 3.49 million aspect-opinion tuples (compared to 3.16 million from a vanilla BERT baseline). It proved exceptionally capable at capturing fine-grained, highly specific customer experiences (extracting 228x more price-nuanced tuples like ($50, parking)) and identifying critical phrases like "hidden, fees" that the baseline completely missed.

Limitations and Future Work

The primary trade-off in Snippext is computational complexity. The label-guessing mechanism in MixMatchNL requires multiple forward passes per unlabeled example, making the training process significantly longer than standard fine-tuning.

For future work, the authors plan to explore optimization strategies to mitigate this overhead, including multitask learning and active learning to further drive down the labeled data requirements.

Summary updated

Abstract

Online services are interested in solutions to opinion mining, which is the problem of extracting aspects, opinions, and sentiments from text. One method to mine opinions is to leverage the recent success of pre-trained language models which can be fine-tuned to obtain high-quality extractions from reviews. However, fine-tuning language models still requires a non-trivial amount of training data. In this paper, we study the problem of how to significantly reduce the amount of labeled training data required in fine-tuning language models for opinion mining. We describe Snippext, an opinion mining system developed over a language model that is fine-tuned through semi-supervised learning with augmented data. A novelty of Snippext is its clever use of a two-prong approach to achieve state-of-the-art (SOTA) performance with little labeled training data through: (1) data augmentation to automatically generate more labeled training data from existing ones, and (2) a semi-supervised learning technique to leverage the massive amount of unlabeled data in addition to the (limited amount of) labeled data. We show with extensive experiments that Snippext performs comparably and can even exceed previous SOTA results on several opinion mining tasks with only half the training data required. Furthermore, it achieves new SOTA results when all training data are leveraged. By comparison to a baseline pipeline, we found that Snippext extracts significantly more fine-grained opinions which enable new opportunities of downstream applications.
BibTeX
@misc{miao2020snippext,
  author = {Zhengjie Miao and Yuliang Li and Xiaolan Wang and Wang-Chiew Tan},
  title = {{Snippext: Semi-supervised Opinion Mining with Augmented Data}},
  year = {2020},
  eprint = {2002.03049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03049},
}

Loading related content...

Figure