A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis

Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong

Abstract

Sentiment analysis is a critical task in natural language processing, often addressed using pre-trained language models. While fine-tuning these models with task-specific layers is standard practice, performance often degrades when training data is scarce. In this paper, we focus on Aspect-Based Sentiment Analysis (ABSA)—extracting aspect terms, categories, and predicting polarities—specifically in few-shot settings. We propose reformulating ABSA from a classification problem into a sequence generation task using a generative language model (GPT-2). By treating extraction and prediction as language generation, the model avoids the need for task-specific layers. Our evaluation shows that this generative approach significantly outperforms BERT-based state-of-the-art models in both few-shot and full-shot settings, notably reducing model variance. Furthermore, the generative framework naturally supports joint and multi-task learning, leading to further performance gains and demonstrating robustness across related tasks like intent detection.

Introduction

Traditional Sentiment Analysis typically classifies a text as positive, negative, or neutral. However, Aspect-Based Sentiment Analysis (ABSA) requires a more granular understanding. For instance, in the sentence, "The food was delicious, but the service was incredibly slow," a model must identify two distinct aspects ("food" and "service") and assign them opposing polarities.

The prevailing paradigm for ABSA involves adding task-specific classification "heads" onto bidirectional models like BERT. While effective with large datasets, this approach falters in "few-shot" scenarios where only a handful of examples are available. These models often suffer from high instability—where small changes in training data or random seeds lead to wildly different results—and require distinct architectures for different sub-tasks. This research explores whether a generative approach can overcome these limitations.

Methodology: ABSA as Sequence Generation

The core innovation of this work is the reformulation of ABSA as a translation-like task. Instead of predicting labels for specific spans of text, the model is trained to generate a structured string that describes the aspects and sentiments found in the input.

The Generative Framework

Using GPT-2, a unidirectional generative model, the researchers convert the input review into a serialized output. For example:

  • Input: [review] The pizza was great [endofreview]
  • Output: [term] pizza positive [endofterm]

This approach eliminates the need for task-specific layers. The model uses the same vocabulary and output layer it was pre-trained with, leveraging its inherent understanding of language structure to format the results.

Joint and Multi-task Learning

Because the output is simply a sequence of tokens, the model can handle multiple objectives simultaneously. It can extract aspect terms, identify categories, and predict polarities in a single forward pass by appending more information to the target string. This unified approach contrasts with traditional methods that often require separate models or complex multi-head architectures for joint tasks.

Key Insights

1. Denser Supervision

In a standard classification setup, the model receives a single signal (the label) for the entire input. In the generative setup, the model calculates a loss for every token it generates. This "denser" supervision provides more feedback per training example, which is particularly beneficial when data is limited.

2. Stability and Reduced Variance

Few-shot learning is notoriously unstable. The researchers found that the generative approach significantly reduced the standard deviation of results compared to BERT-PT. By treating the task as language generation, the model becomes more reliable and less sensitive to the specific examples chosen for the few-shot set.

3. Synergistic Multi-tasking

Interestingly, training the model to perform term extraction and polarity prediction simultaneously actually improved its performance on the individual sub-tasks. Identifying the aspect term provides the necessary context for the model to "focus" its attention on the relevant part of the sentence when determining sentiment.

Experimental Results

The model was evaluated on SemEval 2014 and 2016 (Restaurant and Laptop domains), as well as general sentiment (SST-2/5) and intent detection (OOS) datasets.

  • Few-Shot Performance: On the SemEval 2014 restaurant dataset using only 1% of the training data (approx. 20 examples), the GPT-2 based model outperformed the BERT-PT baseline by 9 points in average accuracy.
  • Variance Reduction: In the same 1% data setting, the generative model reduced the standard deviation of results by 6.2 points, demonstrating superior stability.
  • Full-Shot and Scaling: The approach remained competitive in full-data settings. Furthermore, moving from GPT-2 base to GPT-2 medium yielded additional performance gains, suggesting that the method scales well with model size.
  • Noise Robustness: The model also showed superior performance on SST and OOS datasets, indicating that the generative reformulation is a robust strategy for various few-shot NLP challenges beyond just ABSA.

Discussion and Limitations

While the generative approach offers significant advantages in data efficiency and stability, it introduces new trade-offs. Because GPT-2 is auto-regressive, it generates tokens one by one, making inference slower than traditional classification-head models. Additionally, while the model excels at standard ABSA, it still faces challenges with highly complex linguistic structures, such as sentences with multiple conflicting polarities buried in nested clauses.

Future work may involve scaling this approach to even larger language models and optimizing inference speeds to make generative ABSA more viable for high-throughput real-world applications.

Summary updated

Abstract

Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-task settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-task settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST- and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.
BibTeX
@inproceedings{hosseiniasl2022generative,
  author = {Ehsan Hosseini-Asl and Wenhao Liu and Caiming Xiong},
  title = {{A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=2Ksc_Nr-0p},
}
Figure