Abstract
Sentiment analysis is a critical task in natural language processing, often addressed using pre-trained language models. While fine-tuning these models with task-specific layers is standard practice, performance often degrades when training data is scarce. In this paper, we focus on Aspect-Based Sentiment Analysis (ABSA)—extracting aspect terms, categories, and predicting polarities—specifically in few-shot settings. We propose reformulating ABSA from a classification problem into a sequence generation task using a generative language model (GPT-2). By treating extraction and prediction as language generation, the model avoids the need for task-specific layers. Our evaluation shows that this generative approach significantly outperforms BERT-based state-of-the-art models in both few-shot and full-shot settings, notably reducing model variance. Furthermore, the generative framework naturally supports joint and multi-task learning, leading to further performance gains and demonstrating robustness across related tasks like intent detection.
Introduction
Traditional Sentiment Analysis typically classifies a text as positive, negative, or neutral. However, Aspect-Based Sentiment Analysis (ABSA) requires a more granular understanding. For instance, in the sentence, "The food was delicious, but the service was incredibly slow," a model must identify two distinct aspects ("food" and "service") and assign them opposing polarities.
The prevailing paradigm for ABSA involves adding task-specific classification "heads" onto bidirectional models like BERT. While effective with large datasets, this approach falters in "few-shot" scenarios where only a handful of examples are available. These models often suffer from high instability—where small changes in training data or random seeds lead to wildly different results—and require distinct architectures for different sub-tasks. This research explores whether a generative approach can overcome these limitations.
Methodology: ABSA as Sequence Generation
The core innovation of this work is the reformulation of ABSA as a translation-like task. Instead of predicting labels for specific spans of text, the model is trained to generate a structured string that describes the aspects and sentiments found in the input.
The Generative Framework
Using GPT-2, a unidirectional generative model, the researchers convert the input review into a serialized output. For example:
- Input:
[review] The pizza was great [endofreview] - Output:
[term] pizza positive [endofterm]
This approach eliminates the need for task-specific layers. The model uses the same vocabulary and output layer it was pre-trained with, leveraging its inherent understanding of language structure to format the results.
Joint and Multi-task Learning
Because the output is simply a sequence of tokens, the model can handle multiple objectives simultaneously. It can extract aspect terms, identify categories, and predict polarities in a single forward pass by appending more information to the target string. This unified approach contrasts with traditional methods that often require separate models or complex multi-head architectures for joint tasks.
Key Insights
1. Denser Supervision
In a standard classification setup, the model receives a single signal (the label) for the entire input. In the generative setup, the model calculates a loss for every token it generates. This "denser" supervision provides more feedback per training example, which is particularly beneficial when data is limited.
2. Stability and Reduced Variance
Few-shot learning is notoriously unstable. The researchers found that the generative approach significantly reduced the standard deviation of results compared to BERT-PT. By treating the task as language generation, the model becomes more reliable and less sensitive to the specific examples chosen for the few-shot set.
3. Synergistic Multi-tasking
Interestingly, training the model to perform term extraction and polarity prediction simultaneously actually improved its performance on the individual sub-tasks. Identifying the aspect term provides the necessary context for the model to "focus" its attention on the relevant part of the sentence when determining sentiment.
Experimental Results
The model was evaluated on SemEval 2014 and 2016 (Restaurant and Laptop domains), as well as general sentiment (SST-2/5) and intent detection (OOS) datasets.
- Few-Shot Performance: On the SemEval 2014 restaurant dataset using only 1% of the training data (approx. 20 examples), the GPT-2 based model outperformed the BERT-PT baseline by 9 points in average accuracy.
- Variance Reduction: In the same 1% data setting, the generative model reduced the standard deviation of results by 6.2 points, demonstrating superior stability.
- Full-Shot and Scaling: The approach remained competitive in full-data settings. Furthermore, moving from GPT-2 base to GPT-2 medium yielded additional performance gains, suggesting that the method scales well with model size.
- Noise Robustness: The model also showed superior performance on SST and OOS datasets, indicating that the generative reformulation is a robust strategy for various few-shot NLP challenges beyond just ABSA.
Discussion and Limitations
While the generative approach offers significant advantages in data efficiency and stability, it introduces new trade-offs. Because GPT-2 is auto-regressive, it generates tokens one by one, making inference slower than traditional classification-head models. Additionally, while the model excels at standard ABSA, it still faces challenges with highly complex linguistic structures, such as sentences with multiple conflicting polarities buried in nested clauses.
Future work may involve scaling this approach to even larger language models and optimizing inference speeds to make generative ABSA more viable for high-throughput real-world applications.
Summary updated