Overview of the SIGHAN 2024 shared task for Chinese dimensional aspect-based sentiment analysis

Lung-Hao Lee, Liang-Chih Yu, Suge Wang, Jian LiaoACL 2024

TL;DR

This paper introduces the SIGHAN 2024 shared task for Chinese dimensional Aspect-Based Sentiment Analysis (ABSA), moving beyond discrete sentiment labels to continuous valence-arousal scores. The task aims to advance fine-grained sentiment understanding by organizing three subtasks: intensity prediction, triplet extraction, and quadruple extraction, providing a new benchmark for this challenging NLP problem.

What Problem Does This Paper Address?

Traditional Aspect-Based Sentiment Analysis (ABSA) typically classifies sentiment into discrete categories like positive, neutral, or negative. However, human emotions are far more nuanced and can be better represented by continuous values along dimensions like valence (pleasantness) and arousal (excitement). This paper addresses the need for more sophisticated sentiment analysis by focusing on dimensional ABSA, specifically for the Chinese language. The challenge lies in accurately capturing these continuous emotional states associated with specific aspects within text, which requires more advanced modeling than traditional ABSA.

Methods

The SIGHAN 2024 shared task was organized into three subtasks to tackle dimensional ABSA:

  • Subtask 1: Intensity Prediction: Systems are given a sentence and a specific aspect, and they must predict the sentiment intensity as a pair of continuous valence-arousal scores.
  • Subtask 2: Triplet Extraction: Systems receive a sentence and must extract sentiment triplets, each consisting of an aspect term, an opinion term, and their corresponding valence-arousal intensity.
  • Subtask 3: Quadruple Extraction: This subtask extends triplet extraction by requiring systems to also identify an aspect category, forming a quadruple (aspect, category, opinion, intensity).

The task utilized a dataset derived from restaurant reviews, with manual annotation for sentiment elements and valence-arousal ratings. Both traditional and simplified Chinese versions of the data were provided to accommodate different participant preferences.

Datasets

The primary dataset for this shared task was collected from Google Reviews and the PTT online bulletin board system, focusing on restaurant reviews. After cleaning and preprocessing, sentences were manually annotated. The paper also mentions the Chinese EmoBank (Lee et al., 2022) as a supplementary resource for sentiment analysis, providing valence-arousal scores. The dataset was split into training, development, and two mutually exclusive test sets (one for Subtask 1, and another for Subtasks 2 & 3).

Key Insights and Ideas

  • Dimensionality is Key: The paper strongly advocates for dimensional ABSA, arguing that continuous valence-arousal scores offer a richer and more fine-grained representation of sentiment compared to discrete labels.
  • Task Granularity Matters: By breaking down dimensional ABSA into intensity prediction, triplet extraction, and quadruple extraction, the task allows for a comprehensive evaluation of different aspects of sentiment understanding.
  • Data Imbalance is a Challenge: The aspect category distributions in the dataset are imbalanced, mirroring findings from previous ABSA benchmarks like SemEval-2016. This highlights a persistent challenge in ABSA research.
  • Aspects vs. Opinions: Analysis of the test set revealed that aspects tend to have a more centered distribution (users often discuss similar aspects), while opinion terms exhibit a diverse distribution, indicating a wider range of expressions for user feelings.

Key Results

The SIGHAN 2024 shared task saw 214 submissions from 61 registered participants. The winning team, HITSZ-HLT, demonstrated superior performance across all three subtasks, achieving the best scores in various metrics. CCIIPLab secured second place overall. The results tables (Tables 4, 5, and 6) provide detailed performance metrics (MAE, PCC, F1-score) for each subtask, showcasing the competitive landscape and the effectiveness of different system architectures, including those leveraging pre-trained language models (PLMs) and large language models (LLMs).

Limitations and Future Work

The paper acknowledges that the aspect category distributions are imbalanced, which is a common issue in ABSA datasets. Future work will focus on developing more advanced Chinese dimensional ABSA models and building new language resources to expand this research area, particularly for reviews in other domains. The collected datasets and evaluation scripts are made publicly available to foster further research.

Summary updated

BibTeX
@inproceedings{lee2024overview,
  author = {Lung-Hao Lee and Liang-Chih Yu and Suge Wang and Jian Liao},
  title = {{Overview of the SIGHAN 2024 shared task for Chinese dimensional aspect-based sentiment analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure