Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

Xavier Glorot, Antoine Bordes, Yoshua Bengio

Introduction

Sentiment analysis models often struggle when applied to new domains. A model trained to detect positive sentiment in DVD reviews might look for words like "thrilling" or "hilarious," but these words are useless for evaluating kitchen appliances, where "reliable" or "malfunctioning" are more relevant. Because annotating training data for every conceivable product category is prohibitively expensive, researchers use domain adaptation: training a system on a labeled "source" domain and deploying it on an unlabeled "target" domain.

The core challenge is finding a way to bridge the distribution gap between these domains so the classifier can recognize shared, underlying concepts of sentiment, rather than just memorizing domain-specific buzzwords. To tackle this, the authors propose an unsupervised deep learning approach using Stacked Denoising Auto-encoders (SDAs) to learn a shared, high-level feature representation across all domains. By training a simple linear classifier on these learned features, they achieve state-of-the-art transfer performance on a standard 4-domain benchmark and successfully scale to a massive 22-domain dataset.

Methods

The authors use a two-step Deep Learning approach based on Stacked Denoising Auto-encoders (SDAs) to process text represented as bag-of-words (top 5000 unigrams and bigrams).

  1. Unsupervised Feature Extraction: They train an SDA on the unlabeled text from all available domains. The auto-encoder is trained to reconstruct the original input from a stochastically corrupted version of it (using masking noise). A key novelty here is the use of rectifier units (max(0, x)) in the code layer, which naturally induces a sparse representation.
  2. Supervised Classification: Once the SDA learns a high-level, cross-domain representation, the labeled source domain data is passed through the network to extract these new features. A standard linear Support Vector Machine (SVM) is then trained on this transformed data and evaluated on the target domain.

Unlike previous methods that rely on linear projections (like Structural Correspondence Learning), the SDA learns a highly non-linear mapping. Furthermore, it leverages unlabeled data from all domains simultaneously, rather than just looking at a single source-target pair.

Datasets

  • Amazon 4-Domain Benchmark: A standard, smaller dataset containing reviews for Books, DVDs, Electronics, and Kitchen appliances. It is perfectly balanced (1000 positive and 1000 negative reviews per domain) with a few thousand unlabeled examples.
  • Large-Scale Amazon Dataset: A massive, highly unbalanced dataset introduced for scale testing in this paper. It contains over 340,000 reviews across 22 different product types with varying ratios of positive to negative examples.

Key Insights and Ideas

  • Disentangling Factors of Variation: The deep learning model successfully separates domain-specific vocabulary from general sentiment polarity. The authors found that individual learned features tend to specialize in either recognizing the domain or recognizing the sentiment, but rarely both.
Feature specialization in raw data vs. SDAsh representations
  • The A-Distance Anomaly: Previous domain adaptation theory suggested that for good transfer, the source and target domains should be hard to distinguish in the feature space (low A-distance). Surprisingly, the SDA features actually made the domains easier to distinguish (higher A-distance), yet transfer performance improved drastically. This implies that disentangling features is more important than merely blurring domains together.
  • Negative Transfer Loss: In some cases, the learned representations were so robust that an SVM trained on a different domain actually outperformed an SVM trained directly on the target domain's raw data.
  • Global Unsupervised Pre-training: Using unlabeled data from all 22 domains to train the auto-encoder yielded better representations than just using the specific source and target domains.

Key Results

  • On the 4-domain benchmark, the SDA approach outperformed existing state-of-the-art methods (SCL, SFA, MCT) in 11 out of 12 source-target transfer scenarios.
Transfer loss across 12 domain adaptation scenarios
  • On the massive 22-domain dataset, the baseline linear SVM had an average transfer error of 14.5%. A 3-layer SDA (SDAsh3) reduced this error significantly to 10.9%.
  • Depth matters for complex distributions: On the 22-domain dataset, a 1-layer SDA achieved an 11.5% error, while the 3-layer SDA pushed it down to 10.9%. This proves that deeper architectures capture better hierarchical representations for large-scale transfer.

Limitations and Future Work

  • Limitations: The paper notes that a single-layer architecture is insufficient for capturing the complexity of the large-scale 22-domain dataset; deeper networks (3 layers) are strictly required for optimal performance, which increases computational cost. Additionally, the bag-of-words preprocessing ignores word order and deeper linguistic structures.
  • Future Work: The authors suggest that the surprising increase in the proxy A-distance (domain distinguishability) warrants further theoretical investigation. Understanding exactly how deep networks disentangle domain identity from task labels could lead to better bounds and algorithms for domain adaptation.

Summary updated

Abstract

The exponential increase in the availability of online reviews and recommendations makes sentiment classification an interesting topic in academic and industrial research. Reviews can span so many different domains that it is difficult to gather annotated training data for all of them. Hence, this paper studies the problem of domain adaptation for sentiment classifiers, hereby a system is trained on labeled reviews from one source domain but is meant to be deployed on another. We propose a deep learning approach which learns to extract a meaningful representation for each review in an unsupervised fashion. Sentiment classifiers trained with this high-level feature representation clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products. Furthermore, this method scales well and allowed us to successfully perform domain adaptation on a larger industrial-strength dataset of 22 domains.
BibTeX
@inproceedings{glorot2011domain,
  author = {Xavier Glorot and Antoine Bordes and Yoshua Bengio},
  title = {{Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach}},
  booktitle = {International Conference on Machine Learning},
  year = {2011},
  url = {https://openreview.net/forum?id=BkZFVibd-S},
}
Figure