Biased Representation Learning for Domain Adaptation

Fei Huang, Alexander YatesEMNLP 2012 · OpenReview

TL;DR

This paper tackles the challenge of making NLP models generalize to new domains by introducing "biased representation learning." They propose a framework that allows incorporating expert knowledge into unsupervised feature learning, significantly improving performance on domain adaptation tasks compared to existing methods.

What Problem Does This Paper Address?

Supervised NLP systems excel within their training domains but often see a sharp performance drop when applied to new, out-of-domain data. This is largely due to reliance on lexical features that can differ significantly across domains, and also changes in how words relate to labels. While unsupervised representation learning offers a way to learn features that generalize, these methods typically lack a mechanism to incorporate valuable human expertise. This paper aims to bridge this gap by developing a representation learning approach that can be guided by expert knowledge, leading to more robust and adaptable models.

Methods

The core of this paper is the development of a framework for "biased representation learning" within the context of sequence labeling tasks, specifically using Hidden Markov Models (HMMs). They formalize the representation learning task and argue it's computationally intractable in its general form. To overcome this, they leverage the Posterior Regularization (PR) framework, which modifies the EM algorithm to incorporate constraints or "biases" into the learning objective.

They investigate three types of biases:

  • Task-specific bias (HMM+E): Encourages learned representations to be predictive of the labels in the training data, measured by conditional entropy.
  • Domain-independent bias (HMM+D): Aims to minimize the distance between the feature distributions of the source and target domains, promoting generalization.
  • Multi-dimensional bias (I-HMM+D+E, P-HMM+D+E): Explores learning representations where different dimensions are independent, inspired by Factorial HMMs but with a more efficient approach. They also introduce a bias to encourage different layers in multi-layer models to learn distinct features.

The PR framework allows these biases to be incorporated as penalty terms in the objective function, balanced against the standard marginal log-likelihood objective. They also address the tractability of these biases for structured models by using approximations and efficient inference techniques.

Datasets

  • Chinese POS tagging: The UCLA Corpus of Written Chinese (part of the Lancaster Corpus of Mandarin Chinese - LCMC). They use the 'news' genre as the source domain and randomly select 20% of other genres for labeled test data.
  • English Named Entity Recognition (NER): CONLL 2003 shared task data for labeled training, and MUC7 formal run test data. For NER, they manually labeled WSJ sentences for person, organization, and location tags.

Key Insights and Ideas

  • Representation learning is computationally intractable: The paper formally states the representation learning task and highlights its inherent difficulty, necessitating the use of prior knowledge.
  • Expert knowledge is crucial for generalization: Simply learning features from unlabeled data isn't enough; guiding the learning process with domain or task-specific insights is key to improving out-of-domain performance.
  • Biases can significantly improve domain adaptation: The proposed biased representation learning framework demonstrably outperforms unbiased methods.
  • Task-specific bias needs control: While the entropy bias (task-specific) helps when aligned with the classifier's task, it can hurt performance if not combined with domain-independence, suggesting a need for balance.
  • Multi-dimensional representations offer benefits: Learning representations with independent dimensions can further boost performance, especially when combined with other biases.
  • PR framework is flexible: Posterior Regularization proves to be a powerful tool for incorporating various types of constraints into structured prediction models.

Key Results

  • Significant error reduction: The biased learners achieved a relative reduction in error of more than 16% on both POS tagging and NER tasks compared to existing state-of-the-art representation learning techniques.
  • Chinese POS tagging:
    • The best model (P-HMM+D+E) outperformed the Stanford tagger (state-of-the-art in-domain) on 10 out of 12 target domains.
  • It achieved a 2.6% average improvement over the unbiased HMM and a 1.7% improvement over HMM+D+E.
  • English NER:
    • The best model (P-HMM+D+E) outperformed the unbiased HMM by 3.6% and I-HMM+D+E by 1.6%.
  • Task-specific bias effectiveness: When the entropy bias was aligned with the classifier's task (e.g., POS bias for POS tagging), performance improved by about 1.2% in accuracy (POS) and 1.7% in F1 score (NER). However, switching representations hurt performance.

Limitations and Future Work

  • Intractability of general representation learning: The core problem remains computationally intractable, necessitating the use of biases and approximations.
  • Approximations for distance metric: The paper uses a Euclidean distance approximation for the domain distance metric, which is a simplification of the theoretically proposed d₁ divergence.
  • Parameter tuning: The performance relies on careful tuning of bias parameters (σ and Aii), which is done via grid search on development data.
  • Future Work:
    • Identifying further effective and tractable biases.
  • Extending the approach beyond sequence labeling to other NLP tasks.
  • Exploring more sophisticated multi-dimensional representation learning techniques.

Summary updated

Abstract

Representation learning is a promising technique for discovering features that allow supervised classifiers to generalize from a source domain dataset to arbitrary new domains. We present a novel, formal statement of the representation learning task. We argue that because the task is computationally intractable in general, it is important for a representation learner to be able to incorporate expert knowledge during its search for helpful features. Leveraging the Posterior Regularization framework, we develop an architecture for incorporating biases into representation learning. We investigate three types of biases, and experiments on two domain adaptation tasks show that our biased learners identify significantly better sets of features than unbiased learners, resulting in a relative reduction in error of more than 16% for both tasks, with respect to existing state-of-the-art representation learning techniques.
BibTeX
@inproceedings{huang2012biased,
  author = {Fei Huang and Alexander Yates},
  title = {{Biased Representation Learning for Domain Adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2012},
  url = {https://openreview.net/forum?id=ry-nfGGuWS},
}
Figure