TL;DR
This paper introduces a novel two-stage "Coarse-to-Fine In-context Learning" (CFICL) method to improve fine-grained sentiment intensity prediction for restaurant reviews. By first using fixed examples to enhance a large language model's (LLM) understanding and then refining predictions with similarity-based example selection, the approach significantly boosts accuracy and consistency in predicting valence and arousal scores for aspect terms.
What Problem Does This Paper Address?
Aspect-Based Sentiment Analysis (ABSA) aims to identify specific aspects within text (like "food" or "service" in a restaurant review) and determine the sentiment expressed towards them. Traditional ABSA often focuses on discrete sentiment categories (positive, negative, neutral). However, a more nuanced approach, Dimensional ABSA (dimABSA), seeks to predict sentiment intensity along continuous scales, specifically Valence (pleasantness) and Arousal (excitement/calmness). This task is challenging because it requires capturing subtle emotional nuances and ensuring consistency between these two dimensions. The SIGHAN-2024 dimABSA task specifically focuses on this fine-grained prediction for Chinese restaurant reviews, demanding accurate scores for both valence and arousal for each identified aspect.
Methods
The authors propose a Coarse-to-Fine In-context Learning (CFICL) method built upon the Baichuan2-7B large language model. The approach involves two main stages:
- Fixed In-context Learning Stage (Coarse-grained):
- The model is initially trained using a few-shot learning approach with a small set of fixed, pre-selected in-context examples.
- These examples, along with a prompt template, help the model learn task-specific features and enhance its ability to recognize sentiment elements.
- This stage produces preliminary sentiment quadruplet predictions (aspect, category, opinion, intensity) for the test data.
- Example Retrieval Enhancement Stage (Fine-grained):
- To improve accuracy and granularity, this stage leverages the preliminary predictions from the first stage.
- A BERT model is used to encode the "Opinion" field of each data point.
- Cosine similarity is calculated between the opinion encodings of test data and training data to identify the most similar training examples.
- These highly similar training examples, which include opinion fields, their scores, and related opinion words with average scores, are then used as new, optimized in-context examples.
- Crucially, candidate examples are filtered based on predicted Valence scores to ensure consistency with the test data's emotional polarity.
- Finally, these refined in-context examples are fed back into the model along with the test data for re-prediction, yielding optimized quadruple results.
Datasets
- DimABSA Dataset: Provided by the SIGHAN-2024 workshop organizers. It consists of 6000 training samples, 100 validation samples, and 2000 test samples of restaurant reviews. This dataset is specifically designed for Chinese dimensional ABSA and requires valence and arousal scores for each aspect term.
Key Insights and Ideas
- The power of two-stage learning: The paper demonstrates that a sequential approach, moving from a coarse initial prediction to a fine-grained refinement, is more effective than a single-stage method.
- Similarity-based example selection is key: Simply using fixed examples is insufficient. Dynamically selecting the most relevant training examples based on semantic similarity to the test data significantly improves prediction quality.
- Opinion field is a strong signal: Encoding and leveraging the "Opinion" field for similarity matching proves to be a highly effective strategy for retrieving relevant context.
- Valence filtering for consistency: Filtering candidate examples based on valence scores helps maintain emotional polarity consistency, which is crucial for accurate dimensional sentiment prediction.
- LLMs benefit from tailored context: Even powerful LLMs like Baichuan2-7B can achieve better performance on specialized tasks like dimABSA when provided with carefully curated and dynamically selected in-context examples.
Key Results
- The proposed Coarse-to-Fine In-context Learning (CFICL) method achieved third place in the SIGHAN-2024 dimABSA task (specifically for triplet and quadruple extraction subtasks).
- The method significantly enhances prediction accuracy and granularity for both valence and arousal dimensions.
- It effectively reduces sentiment polarity bias.
- Experimental results show that the optimized prompt and example selection in the second stage lead to more reasonable and consistent valence and arousal scores. For instance, in a case study, scores for "my love" improved from 6.5/6.0 to 7.5/7.25 with optimized examples, and subsequent re-predictions for a related phrase adjusted to a more fitting 7.0/7.0.
Limitations and Future Work
- Limited scope of optimization: The current method primarily focuses on improving sentiment intensity prediction (valence and arousal) and does not further optimize the extraction of Aspect and Category fields.
- Dependency on similarity measure: The effectiveness of the second stage relies heavily on the accuracy of the similarity measure between the Opinion field and the training data. Error propagation from this step requires further investigation.
- High computational demands: The method, especially the large-scale data training and optimization, requires significant computational resources, which could limit its widespread adoption.
- Future work: The authors suggest further research into optimizing Aspect and Category extraction and exploring more robust similarity measures to mitigate error propagation. They also acknowledge the need for more efficient computational approaches.
Summary updated