Abstract
Traditional approaches to Aspect-Based Sentiment Analysis (ABSA) rely on discriminative classifiers with task-specific heads, which often struggle in low-resource settings. This paper introduces a generative reformulation of ABSA, treating it as a sequence generation task. By fine-tuning generative models like GPT-2 to produce serialized strings of aspects and polarities, we achieve state-of-the-art performance. Our approach is particularly effective in few-shot scenarios, significantly outperforming BERT-based classifiers and demonstrating greater robustness to limited training data.
Introduction
Sentiment analysis has evolved beyond simple document-level classification. Aspect-Based Sentiment Analysis (ABSA) requires identifying specific aspect terms (e.g., "pizza"), their categories (e.g., "food"), and the associated sentiment polarity (e.g., "positive"). For instance, in the sentence "The food was delicious, but the service was slow," a model must extract two distinct opinions.
Historically, researchers have addressed this by adding specialized classification layers on top of encoders like BERT. However, these methods are often "data-hungry," requiring large labeled datasets to converge. Furthermore, the rigid separation of sub-tasks—extraction, categorization, and polarity prediction—prevents models from fully leveraging the relationships between these components.
Methods: ABSA as Language Generation
We propose a paradigm shift: treating ABSA as a text-to-text generation problem. Instead of predicting discrete labels, a generative model (GPT-2) is fine-tuned to "write out" the analysis.
Given an input review, the model generates a structured string:
- Input:
[review] The pizza was great but expensive [endofreview] - Generated Output:
[term] pizza positive, price negative [endofterm]
Technical Innovations
- Unified Architecture: By using a text-to-text format, the model employs the same weights for both extraction and classification, eliminating the need for custom "heads."
- Joint & Multi-task Training: The model can be trained to generate terms and categories simultaneously. This multi-tasking allows the model to learn that "pizza" belongs to the "food" category, which in turn improves its ability to identify "pizza" as an aspect term.
- Generative Supervision: Unlike classifiers that receive a signal from a single label, generative models learn from every token in the output sequence. This provides a much richer supervision signal, especially beneficial when data is scarce.
Key Insights and Ideas
The generative approach offers a distinct "Generative Advantage" in few-shot settings. When trained on only 1% of available data (approximately 20 examples), the generative model remains robust, showing significantly lower variance across random seeds compared to BERT-based models, which often fail to converge or produce erratic results.
Furthermore, we found that the magic lies in the generative loss itself. Using GPT-2 as a standard classifier (using only the last token's hidden state) results in performance inferior to BERT. Forcing the model to predict the next word in a descriptive sentence helps it internalize the underlying sentiment structure more effectively.
Key Results
Our evaluation across standard benchmarks like Semeval 2014 and 2016 demonstrates the superiority of the generative paradigm.
- Few-Shot Dominance: In the restaurant domain with only 1% training data, the GPT-2 based model outperformed the previous state-of-the-art (BERT-PT) by 9 points in accuracy and reduced standard deviation by over 6 points.
- Full-Shot Performance: The generative approach remains competitive even with 100% of the training data, matching or exceeding BERT-based models. This suggests that sequence generation is a fundamentally strong way to model ABSA, not just a workaround for small datasets.
- Robustness: On fine-grained sentiment tasks (like SST-5), the generative model better handled "noisy" boundaries between similar sentiments, such as "somewhat positive" versus "positive."
Limitations and Future Work
Despite its strengths, the generative approach has limitations. Autoregressive generation—producing text token-by-token—is computationally more expensive and slower than the single-pass inference of a classifier. This may pose challenges for real-time applications. Additionally, while the model excels at standard tasks, it can still struggle with highly complex sentences containing multiple conflicting opinions. Future work will explore performance in low-resource languages and methods to optimize inference speed.
Summary updated