Projecting Embeddings for Domain Adaption: Joint Modeling of Sentiment Analysis in Diverse Domains

Jeremy Barnes, Roman Klinger, Sabine Schulte im WaldeCOLING 2018

TL;DR

This paper introduces a novel approach to domain adaptation for sentiment analysis by framing it as an embedding projection task. By learning to project word embeddings from different domains into a shared "bi-domain" space, their model achieves state-of-the-art results, particularly on highly divergent domains where traditional methods struggle.

What Problem Does This Paper Address?

Sentiment analysis models are notoriously sensitive to domain shifts. A classifier trained on movie reviews, for instance, might perform poorly on product reviews or tweets due to differences in vocabulary, writing style, and context. This "domain adaptation" problem is crucial for building robust NLP systems. Existing methods like Structural Correspondence Learning (SCL) and autoencoders often require extensive training or lose information by forcing domain-agnostic representations, leading to performance degradation on dissimilar domains. This paper aims to bridge this gap by preserving domain-specific features while enabling cross-domain transfer.

Methods

The core of this work is a novel embedding projection technique inspired by cross-lingual word embedding methods. The authors propose learning two linear projection matrices, one for the source domain ($M$) and one for the target domain ($M'$). These matrices project word embeddings from their respective domain-specific spaces into a shared "bi-domain" space. This projection is jointly optimized with a sentiment classification objective for the source domain.

The key innovation lies in the dual objective:

  1. Projection Objective: Minimize the mean squared error between projected word vectors from the source and target domains, using a lexicon of aligned words. This encourages the projection to align similar words across domains.
  2. Sentiment Objective: Train a sentiment classifier on the projected source domain embeddings. This forces the projection matrices to learn representations that are not only aligned but also sentiment-aware.

By jointly optimizing these objectives, the model learns to project target domain embeddings in a way that is useful for sentiment classification, even without direct sentiment labels for the target domain.

Datasets

The authors evaluate their approach on several benchmark datasets:

  • Amazon Corpora: Four domains (books, DVD, electronics, kitchen) collected by Blitzer et al. (2007). These are used for training and testing, with a focus on domain adaptation between these categories.
  • SemEval 2013 and 2016: Message polarity classification tasks from Twitter data. These are used to evaluate performance on highly divergent domains compared to the Amazon corpora.

Key Insights and Ideas

  • Embedding Projection as Domain Adaptation: Reframing domain adaptation as an embedding projection task offers a powerful new perspective, allowing for the preservation of domain-specific nuances while achieving cross-domain transfer.
  • Dual Projection Matrices ($M$ and $M'$): The use of separate projection matrices for source and target domains is crucial. While $M$ aligns source embeddings and aids sentiment classification, $M'$ is essential for transferring sentiment information to the target domain, enabling sentiment prediction without direct target labels.
  • Performance on Divergent Domains: The model demonstrates a significant advantage over existing state-of-the-art methods when domains are highly dissimilar, suggesting that preserving domain-specific features through projection is more effective than aggressive domain-agnostic methods.
  • Lexicon Choice Matters: The choice of projection lexicon (aligned words) impacts performance, with frequency-based lexicons performing better on divergent domains and sentiment lexicons on more similar ones. Mutual Information-based lexicons show promise but require careful tuning.

Key Results

  • State-of-the-Art Performance: The proposed model (BLSE) achieves novel state-of-the-art results on 11 out of 20 domain pairs tested.
  • Superiority on Divergent Domains: On the SemEval datasets (which are more divergent from the Amazon corpora), BLSE significantly outperforms baselines like NSCL, MSDA, and NOAD, showing improvements of 8.2, 15.5, and 6.4 F1 points on average, respectively.
  • Comparable Performance on Similar Domains: On more similar domains (e.g., within the Amazon corpora), BLSE performs comparably to or slightly better than state-of-the-art methods like NSCL.
  • Error Analysis: The model shows a better ability to handle minority classes compared to NSCL and MSDA, which sometimes exhibit extreme biases.

Limitations and Future Work

  • Current Disadvantage: The model relies on skip-gram embeddings trained on multiple domains. The authors suggest investigating domain-specific embeddings to potentially improve results further.
  • Future Directions:
    • Multi-Domain Mapping: Extend the model to learn multiple domain mappings simultaneously, enabling zero-shot domain adaptation at scale.
  • Concept Mapping: Develop lexicons that map concepts (e.g., "read" in books to "watch" in movies) rather than just words, potentially using vector algebra to find analogous concepts.
  • Multi-Word Unit Mapping: Extend the approach to map multi-word expressions across domains.
  • Aspect-Level Sentiment: Apply the framework to more granular tasks like aspect-level sentiment analysis, where domain adaptation is particularly challenging.

Summary updated

BibTeX
@inproceedings{barnes2018projecting,
  author = {Jeremy Barnes and Roman Klinger and Sabine Schulte im Walde},
  title = {{Projecting Embeddings for Domain Adaption: Joint Modeling of Sentiment Analysis in Diverse Domains}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure