TL;DR
This paper introduces a novel In-Context Learning (ICL) framework to improve Large Language Models' (LLMs) ability to perform Dimensional Aspect-Based Sentiment Analysis (dimABSA). By developing a smart sample selection method that considers both sentence and aspect similarity, they significantly boost LLM performance on this fine-grained sentiment analysis task.
What Problem Does This Paper Address?
Traditional sentiment analysis often provides a single positive, negative, or neutral label for an entire text. Aspect-Based Sentiment Analysis (ABSA) refines this by identifying specific aspects within a text and their associated sentiment. However, human emotions are more nuanced than discrete labels. This paper focuses on Dimensional Aspect-Based Sentiment Analysis (dimABSA), which aims to capture sentiment along two continuous dimensions: valence (pleasure vs. displeasure) and arousal (intensity of emotion). This provides a much richer understanding of sentiment.
While LLMs have shown promise in various NLP tasks, their application to dimABSA, especially using in-context learning (ICL), has been underexplored. ICL allows LLMs to adapt to new tasks by providing a few examples in the prompt, without requiring model retraining. The challenge lies in selecting the best examples to include in the prompt to maximize performance. Simply picking random examples might not be representative of the target task, leading to suboptimal results. This paper addresses this gap by proposing a more intelligent way to select these in-context examples for dimABSA.
Methods
The core contribution of this paper is a novel In-Context Learning (ICL) structure designed specifically for dimABSA. This structure consists of two main components:
- Task Instruction: A clear description of the dimABSA task, including definitions of valence, arousal, and aspect, along with the required input and output formats.
- Sample Selector (S4): This is the key innovation. Instead of random selection, the authors propose a Semantic Similarity Sample Selector (S4). This method aims to pick the most relevant examples from a training set to include in the prompt for a given test sample.
- Aspect-Aware Similarity: The S4 goes beyond simple sentence similarity. It recognizes that the same word can have different sentiment implications depending on the aspect it describes (e.g., "sour" lemon vs. "sour" milk). Therefore, it calculates similarity by considering both the overall sentence semantics (using BERT embeddings) and the similarity of the aspects present in the sentences.
- Combined Similarity: The final similarity score between a test sample and a training sample is a weighted combination of sentence similarity and aspect similarity.
- Prompt Construction: The top N (they used N=10) most similar samples, selected by S4, are then included in the prompt along with the task instructions and the test sample.
The paper also outlines two settings for using LLMs:
- Zero-shot setting: The LLM is given only the task instruction and the test sample, with no examples.
- Few-shot setting: The LLM is provided with examples, either selected randomly or using the proposed S4 method.
Datasets
The paper uses a dataset provided by the organizer of the SIGHAN-2024 dimABSA task. This dataset contains 3000 sentences, each with one or more aspects annotated with corresponding valence and arousal values ranging from 1 to 9. No new benchmark datasets were introduced.
Key Insights and Ideas
- The Importance of Aspect-Awareness: The paper strongly emphasizes that for dimABSA, simply matching sentence semantics isn't enough. The specific aspect being discussed is crucial for determining sentiment, and the S4 method effectively captures this.
- LLMs Benefit from Smart ICL: The results clearly show that LLMs, even powerful ones like GPT-4, perform significantly better on dimABSA when provided with well-selected in-context examples compared to zero-shot or random selection.
- Valence and Arousal are Distinct: The evaluation metrics (V-MAE, V-PCC, A-MAE, A-PCC) treat valence and arousal separately, highlighting that improving one doesn't automatically improve the other, and that different LLMs might excel at one dimension over the other.
Key Results
The paper presents results comparing three LLMs (qwen-plus, GPT-3.5, and GPT-4) across zero-shot, random selection few-shot, and S4 few-shot settings.
- GPT-4 with S4 Achieves Top Performance: The GPT-4 model using the proposed S4 framework achieved the best performance on Valence MAE (V-MAE), Valence PCC (V-PCC), and Arousal PCC (A-PCC).
- qwen-plus with S4 Excels in Arousal MAE: The qwen-plus model with S4 slightly outperformed GPT-4 on Arousal MAE (A-MAE).
- Significant Improvements: Experimental results indicate that the S4 method significantly improves the sentiment analysis capabilities of LLMs for dimABSA, with statistical significance noted for several comparisons against zero-shot and random selection. For instance, GPT-3.5 w. S4 achieved a V-PCC of 0.890, a substantial improvement over its zero-shot V-PCC of 0.600.
Limitations and Future Work
- Reliance on Proprietary LLMs: A primary limitation acknowledged is the reliance on proprietary LLMs (like GPT-3.5 and GPT-4). This poses challenges for reproducibility, as access to these models can be restricted.
- Future Work: The authors plan to extend their experiments to include a broader range of open-source LLMs (e.g., LLaMA2, LLaMA3) to develop a more performant and generalized approach.
Summary updated