Introduction
Deep learning models typically require vast amounts of labeled data to achieve high performance. While coarse-grained labels (e.g., sentence-level sentiment) are relatively abundant, fine-grained labels (e.g., aspect-level sentiment) are expensive and time-consuming to annotate. This paper introduces a novel transfer learning framework that leverages Expectation Regularization (XR) to bridge the gap between these related tasks. By matching the proportions of labels in a dataset rather than individual labels, the proposed method effectively utilizes large-scale unlabeled data guided by coarse-grained supervision.
Methodology
The core innovation is the use of XR to train models for a target task (Task B) using labels from a related source task (Task A). While a positive sentence (Task A) does not guarantee every aspect within it is positive (Task B), there is a strong statistical correlation.
Expectation Regularization (XR) Framework
Instead of standard cross-entropy loss, which penalizes individual prediction errors, XR penalizes the model if the distribution of its predictions over a data partition does not match an expected proportion. The framework consists of three primary components:
- Transfer via Proportions: A classifier for the source task groups unlabeled data. Expected proportions of the target task labels are then estimated within these groups (e.g., "In sentences labeled 'Positive', we expect 90% of aspects to be positive").
- Stochastic Batched XR: To make XR computationally feasible for deep learning, the authors propose a stochastic approximation where proportions are calculated over small, random batches rather than the entire dataset.
- Fragment Decomposition: For Aspect-Based Sentiment Classification (ABSC), a constituency parse tree heuristic is used to decompose sentences into fragments associated with specific aspects, allowing the model to focus on relevant textual context.
Experimental Results
The authors evaluated their approach on the SemEval-2015 and SemEval-2016 restaurant review datasets. They utilized up to 670,000 sentences from the Yelp Dataset Challenge for sentence-level sentiment and large-scale unlabeled training.
The XR-based models consistently outperformed several state-of-the-art fully supervised models. As shown in the results table below, the BiLSTM-XR model achieved an accuracy of 83.31% on the SemEval-2015 dataset, surpassing previous benchmarks. Furthermore, the method showed significant synergy with BERT; combining XR with BERT fine-tuning led to accuracies of 85.78% (SemEval-15) and 89.57% (SemEval-16), outperforming standard BERT fine-tuning by approximately 4%.
Discussion and Key Insights
A primary insight of this work is that aggregate statistics are often more stable than individual noisy labels. By training on expected proportions, the model learns the underlying signal while remaining robust to outliers. The XR loss acts as a semi-supervised bridge, allowing the model to benefit from hundreds of thousands of unlabeled examples.
Limitations
Despite its success, the approach has limitations. The fragment decomposition relies on heuristic rules and constituency parsers, which may fail on complex sentence structures. Additionally, the method requires a strong statistical correlation between the source and target tasks. The "Neutral" sentiment class remains a challenge due to its lower frequency and the resulting difficulty in accurate proportion estimation.
Conclusion
This research demonstrates that label proportions provide a powerful form of supervision that complements traditional self-supervised pre-training like BERT. The XR framework offers a stable and effective way to perform transfer learning between related tasks, significantly reducing the reliance on expensive fine-grained annotations.
Summary updated