Transfer Learning Between Related Tasks Using Expected Label Proportions

Matan Ben Noach, Yoav Goldberg

Introduction

Deep learning models are notoriously data-hungry, often requiring vast amounts of granularly labeled data to achieve high performance. In many domains, however, obtaining fine-grained labels (e.g., aspect-level sentiment) is significantly more expensive and time-consuming than obtaining coarse-grained labels (e.g., sentence-level sentiment). This paper, presented by Matan Ben Noach and Yoav Goldberg, introduces a novel approach to bridge this gap by leveraging the statistical relationships between related tasks.

The authors adapt Expectation Regularization (XR) to perform transfer learning, allowing a model to learn a target task (Task B) using labels from a related source task (Task A). By introducing a stochastic batched approximation, they make XR viable for large-scale deep learning, achieving state-of-the-art results on aspect-based sentiment analysis with minimal direct supervision.

The Data Bottleneck in Sentiment Analysis

The core challenge addressed is the disparity in labeling effort. While it is relatively easy to determine if an entire review is positive, identifying the sentiment toward specific entities—like "service" or "food"—within that review requires expert annotation.

The authors exploit the fact that these tasks are often correlated: if a sentence is positive, there is a high statistical probability that the individual aspects within it are also positive. By training on these aggregate statistical proportions rather than individual noisy labels, the model can learn the underlying sentiment signal more effectively.

Methodology: Stochastic Batched XR

The primary technical contribution is the application of Expectation Regularization (XR). Unlike standard supervised learning, which minimizes loss on individual point predictions, XR trains the model to ensure its predictions across a set of points match a target distribution (label proportions).

Overview of the Stochastic Batched Expectation Regularization (XR) framework.

The framework consists of three main components:

  1. Transfer-training via XR: A source classifier (Task A) is used to label a large unlabeled corpus. The conditional probability of target labels (Task B) is then estimated given these source labels. For instance, the model might learn that in "Positive" sentences, 90% of aspects are positive.
  2. Stochastic Batched XR: Traditional XR requires calculating expectations over the entire dataset, which is computationally prohibitive for modern deep learning. The authors propose a stochastic version that samples random batches and optimizes the model to match proportions within those batches.
  3. Fragment Decomposition: To apply this to Aspect-Based Sentiment Analysis (ABSA), the authors use constituency parse trees to heuristically break sentences into "fragments" associated with specific aspects, allowing the model to focus on relevant text segments.

Experimental Results

The researchers evaluated their approach using the SemEval 2015 and 2016 benchmarks for restaurant reviews, alongside the Yelp Dataset Challenge for source labels.

Performance comparison of BiLSTM-XR against state-of-the-art supervised and semi-supervised methods.

The proposed method, BiLSTM-XR, consistently outperformed fully supervised baselines:

  • Accuracy: On SemEval-2015, the XR-based model achieved 83.31% accuracy, surpassing the best supervised baseline of 81.67%.
  • Synergy with BERT: The method is additive to modern architectures. When combined with BERT, the model reached 85.78% accuracy on SemEval-2015 and 89.57% on SemEval-2016.
  • Stability: XR training demonstrated higher stability and lower standard deviation across random initializations compared to other semi-supervised methods.

Key Insights and Future Directions

A major takeaway is that aggregate signals can overcome individual noise. While a sentence-level label might be a "noisy" indicator for a specific aspect, it provides a stable indicator for a group of aspects. By training on aggregate proportions, the model avoids being misled by outliers.

Furthermore, the transition from traditional logistic regression to deep neural networks via stochastic batching makes a decade-old statistical technique highly relevant for modern NLP.

Limitations

The primary requirement is a known or estimable correlation between the source and target tasks. If the tasks are not statistically related, the expected proportions become meaningless. Additionally, the reliance on constituency parsing for fragment decomposition may limit applicability in domains or languages where such tools are less accurate.

Summary updated

BibTeX
@inproceedings{noach2019transfer,
  author = {Matan Ben Noach and Yoav Goldberg},
  title = {{Transfer Learning Between Related Tasks Using Expected Label Proportions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1004},
}
Figure