LLaMA-Based Models for Aspect-Based Sentiment Analysis

Jakub Šmíd, Pavel Priban, Pavel KralACL 2024

TL;DR

This paper explores the potential of fine-tuning open-source LLaMA-based large language models (LLMs) for Aspect-Based Sentiment Analysis (ABSA). They found that while LLMs struggle in zero-shot and few-shot settings, fine-tuning, particularly with the Orca 2 model, achieves state-of-the-art results across multiple ABSA tasks and datasets, outperforming previous benchmarks.

What Problem Does This Paper Address?

Aspect-Based Sentiment Analysis (ABSA) is a crucial task in understanding nuanced opinions within text, going beyond simple positive/negative sentiment to identify specific aspects (e.g., "food quality") and their associated opinions (e.g., "delicious"). While large language models (LLMs) have shown promise in various NLP tasks, their performance in complex, compound ABSA tasks often lags behind specialized, fine-tuned models. This paper investigates whether fine-tuning LLMs, specifically open-source LLaMA-based models, can bridge this performance gap and achieve state-of-the-art results in ABSA.

Methods

The researchers evaluated two LLaMA-based models: LLaMA 2 (7B and 13B versions) and Orca 2 (7B and 13B versions). For fine-tuning, they employed QLoRA, a memory-efficient technique that uses 4-bit quantization and LoRA adapters, allowing for fine-tuning on a single GPU. This approach involves a frozen 4-bit quantized backbone LLM with a small set of learnable LoRA weights. They also explored zero-shot and few-shot scenarios, using 4-bit quantization for these experiments as well. Evaluation was primarily based on micro F1-score, with a strict requirement that all elements of a predicted sentiment tuple must exactly match the gold tuple.

Datasets

The study utilized eight English datasets across four ABSA tasks:

  • ASQP (Aspect Sentiment Quadruplet Prediction): Rest15, Rest16
  • ACOS (Aspect Category Opinion Sentiment): ACOS-Rest, ACOS-Lap
  • TASD (Target Aspect Sentiment Detection): Datasets from Xu et al. (2020) and Wan et al. (2020)
  • ASTE (Aspect Sentiment Triplet Extraction): Datasets from Xu et al. (2020) and Wan et al. (2020)

These datasets were chosen to align with previous research for fair comparison. The ACOS datasets focus on implicit aspects and opinions, while ASTE datasets do not include implicit sentiment elements.

Key Insights and Ideas

  • Fine-tuning is Key: The paper strongly suggests that while LLMs have inherent capabilities, fine-tuning them on specific ABSA tasks is crucial for achieving competitive performance. Zero-shot and few-shot performance, while explored, significantly underperforms fine-tuned models.
  • Orca 2 Shines: The Orca 2 model, particularly its 13B version, demonstrated remarkable potential, surpassing state-of-the-art results across all evaluated tasks and datasets. This highlights the effectiveness of Orca 2's enhanced reasoning capabilities.
  • Model Size vs. Sophistication: The results suggest that a more advanced, smaller model (like Orca 2 13B) can outperform larger, less sophisticated models (like LLaMA 2 13B), indicating that architectural improvements and reasoning abilities are highly valuable.
  • TASD Presents a Unique Challenge: The TASD task, which doesn't require predicting opinion terms, showed the most significant improvement with fine-tuned LLMs, suggesting that predicting opinion terms (present in other tasks) is a major hurdle.
  • Dataset Labeling Matters: The error analysis revealed inconsistencies in dataset labels, which could negatively impact model performance and evaluation.

Key Results

  • State-of-the-Art Performance: The fine-tuned Orca 2 13B model achieved state-of-the-art results on all eight datasets and four ABSA tasks, with notable improvements in TASD (3-5% over previous SOTA).
  • Orca 2 vs. LLaMA 2: Orca 2 models consistently outperformed LLaMA 2 models. The Orca 2 7B model even surpassed the larger LLaMA 2 13B model.
  • Zero-Shot/Few-Shot Underperformance: In zero-shot and few-shot scenarios, both LLaMA-based models performed significantly worse than fine-tuned models and even ChatGPT (which, despite its size, underperformed compared to fine-tuned models).
  • Error Analysis: The most challenging sentiment element to predict was the opinion term, followed by the aspect term. Sentiment polarity was generally the easiest to predict, with ACOS-Lap being an exception where aspect category was the most difficult due to its vast number of categories.

Limitations and Future Work

  • Zero-Shot/Few-Shot Limitations: The paper acknowledges the ineffectiveness of LLaMA-based models in zero-shot and few-shot compound ABSA tasks.
  • Language Specificity: The current evaluation is limited to English ABSA tasks, and the performance of these models in non-English languages remains unexplored.
  • Future Directions: Future work could explore other open-source models with different architectures and investigate their performance on ABSA tasks. Further research into improving zero-shot and few-shot capabilities for LLMs in ABSA would also be valuable. The authors also suggest that addressing the challenges in predicting opinion terms and improving dataset quality could lead to further advancements.

Summary updated

BibTeX
@inproceedings{smid2024llama,
  author = {Jakub Šmíd and Pavel Priban and Pavel Kral},
  title = {{LLaMA-Based Models for Aspect-Based Sentiment Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure