Introduction
Deep learning models are notoriously data-hungry, often requiring vast amounts of granularly labeled data to achieve high performance. In many domains, however, obtaining fine-grained labels (e.g., aspect-level sentiment) is significantly more expensive and time-consuming than obtaining coarse-grained labels (e.g., sentence-level sentiment). This paper, presented by Matan Ben Noach and Yoav Goldberg, introduces a novel approach to bridge this gap by leveraging the statistical relationships between related tasks.
The authors adapt Expectation Regularization (XR) to perform transfer learning, allowing a model to learn a target task (Task B) using labels from a related source task (Task A). By introducing a stochastic batched approximation, they make XR viable for large-scale deep learning, achieving state-of-the-art results on aspect-based sentiment analysis with minimal direct supervision.
The Data Bottleneck in Sentiment Analysis
The core challenge addressed is the disparity in labeling effort. While it is relatively easy to determine if an entire review is positive, identifying the sentiment toward specific entities—like "service" or "food"—within that review requires expert annotation.
The authors exploit the fact that these tasks are often correlated: if a sentence is positive, there is a high statistical probability that the individual aspects within it are also positive. By training on these aggregate statistical proportions rather than individual noisy labels, the model can learn the underlying sentiment signal more effectively.
Methodology: Stochastic Batched XR
The primary technical contribution is the application of Expectation Regularization (XR). Unlike standard supervised learning, which minimizes loss on individual point predictions, XR trains the model to ensure its predictions across a set of points match a target distribution (label proportions).
The framework consists of three main components:
- Transfer-training via XR: A source classifier (Task A) is used to label a large unlabeled corpus. The conditional probability of target labels (Task B) is then estimated given these source labels. For instance, the model might learn that in "Positive" sentences, 90% of aspects are positive.
- Stochastic Batched XR: Traditional XR requires calculating expectations over the entire dataset, which is computationally prohibitive for modern deep learning. The authors propose a stochastic version that samples random batches and optimizes the model to match proportions within those batches.
- Fragment Decomposition: To apply this to Aspect-Based Sentiment Analysis (ABSA), the authors use constituency parse trees to heuristically break sentences into "fragments" associated with specific aspects, allowing the model to focus on relevant text segments.
Experimental Results
The researchers evaluated their approach using the SemEval 2015 and 2016 benchmarks for restaurant reviews, alongside the Yelp Dataset Challenge for source labels.
The proposed method, BiLSTM-XR, consistently outperformed fully supervised baselines:
- Accuracy: On SemEval-2015, the XR-based model achieved 83.31% accuracy, surpassing the best supervised baseline of 81.67%.
- Synergy with BERT: The method is additive to modern architectures. When combined with BERT, the model reached 85.78% accuracy on SemEval-2015 and 89.57% on SemEval-2016.
- Stability: XR training demonstrated higher stability and lower standard deviation across random initializations compared to other semi-supervised methods.
Key Insights and Future Directions
A major takeaway is that aggregate signals can overcome individual noise. While a sentence-level label might be a "noisy" indicator for a specific aspect, it provides a stable indicator for a group of aspects. By training on aggregate proportions, the model avoids being misled by outliers.
Furthermore, the transition from traditional logistic regression to deep neural networks via stochastic batching makes a decade-old statistical technique highly relevant for modern NLP.
Limitations
The primary requirement is a known or estimable correlation between the source and target tasks. If the tasks are not statistically related, the expected proportions become meaningless. Additionally, the reliance on constituency parsing for fragment decomposition may limit applicability in domains or languages where such tools are less accurate.
Summary updated