A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis

Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong

Abstract

Traditional approaches to Aspect-Based Sentiment Analysis (ABSA) rely on discriminative classifiers with task-specific heads, which often struggle in low-resource settings. This paper introduces a generative reformulation of ABSA, treating it as a sequence generation task. By fine-tuning generative models like GPT-2 to produce serialized strings of aspects and polarities, we achieve state-of-the-art performance. Our approach is particularly effective in few-shot scenarios, significantly outperforming BERT-based classifiers and demonstrating greater robustness to limited training data.

Introduction

Sentiment analysis has evolved beyond simple document-level classification. Aspect-Based Sentiment Analysis (ABSA) requires identifying specific aspect terms (e.g., "pizza"), their categories (e.g., "food"), and the associated sentiment polarity (e.g., "positive"). For instance, in the sentence "The food was delicious, but the service was slow," a model must extract two distinct opinions.

Historically, researchers have addressed this by adding specialized classification layers on top of encoders like BERT. However, these methods are often "data-hungry," requiring large labeled datasets to converge. Furthermore, the rigid separation of sub-tasks—extraction, categorization, and polarity prediction—prevents models from fully leveraging the relationships between these components.

Methods: ABSA as Language Generation

We propose a paradigm shift: treating ABSA as a text-to-text generation problem. Instead of predicting discrete labels, a generative model (GPT-2) is fine-tuned to "write out" the analysis.

Given an input review, the model generates a structured string:

  • Input: [review] The pizza was great but expensive [endofreview]
  • Generated Output: [term] pizza positive, price negative [endofterm]

Technical Innovations

  1. Unified Architecture: By using a text-to-text format, the model employs the same weights for both extraction and classification, eliminating the need for custom "heads."
  2. Joint & Multi-task Training: The model can be trained to generate terms and categories simultaneously. This multi-tasking allows the model to learn that "pizza" belongs to the "food" category, which in turn improves its ability to identify "pizza" as an aspect term.
  3. Generative Supervision: Unlike classifiers that receive a signal from a single label, generative models learn from every token in the output sequence. This provides a much richer supervision signal, especially beneficial when data is scarce.

Key Insights and Ideas

The generative approach offers a distinct "Generative Advantage" in few-shot settings. When trained on only 1% of available data (approximately 20 examples), the generative model remains robust, showing significantly lower variance across random seeds compared to BERT-based models, which often fail to converge or produce erratic results.

Performance comparison on Semeval 2016 benchmarks, highlighting the generative model's advantage in few-shot settings.

Furthermore, we found that the magic lies in the generative loss itself. Using GPT-2 as a standard classifier (using only the last token's hidden state) results in performance inferior to BERT. Forcing the model to predict the next word in a descriptive sentence helps it internalize the underlying sentiment structure more effectively.

Key Results

Our evaluation across standard benchmarks like Semeval 2014 and 2016 demonstrates the superiority of the generative paradigm.

Performance comparison on Semeval 2014 benchmarks across varying training data sizes.
  • Few-Shot Dominance: In the restaurant domain with only 1% training data, the GPT-2 based model outperformed the previous state-of-the-art (BERT-PT) by 9 points in accuracy and reduced standard deviation by over 6 points.
  • Full-Shot Performance: The generative approach remains competitive even with 100% of the training data, matching or exceeding BERT-based models. This suggests that sequence generation is a fundamentally strong way to model ABSA, not just a workaround for small datasets.
  • Robustness: On fine-grained sentiment tasks (like SST-5), the generative model better handled "noisy" boundaries between similar sentiments, such as "somewhat positive" versus "positive."

Limitations and Future Work

Despite its strengths, the generative approach has limitations. Autoregressive generation—producing text token-by-token—is computationally more expensive and slower than the single-pass inference of a classifier. This may pose challenges for real-time applications. Additionally, while the model excels at standard tasks, it can still struggle with highly complex sentences containing multiple conflicting opinions. Future work will explore performance in low-resource languages and methods to optimize inference speed.

Summary updated

Abstract

Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-task settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-task settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST- and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.
BibTeX
@inproceedings{hosseiniasl2022generative,
  author = {Ehsan Hosseini-Asl and Wenhao Liu and Caiming Xiong},
  title = {{A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=2Ksc_Nr-0p},
}
Figure