ZHIXIAOBAO at SemEval-2022 Task 10: Apporoaching Structured Sentiment with Graph Parsing

Yangkun Lin, Chen Liang, Jing Xu, Chong Yang, Yongliang WangNAACL 2022

TL;DR

This paper presents a novel approach to structured sentiment analysis by framing it as a graph parsing problem. By leveraging pre-trained language models and introducing discrete cross-attention and suffix masking techniques, the ZHIXIAOBAO system achieved top performance in the SemEval-2022 Task 10, outperforming 31 other teams.

What Problem Does This Paper Address?

Traditional sentiment analysis often focuses on identifying the overall sentiment of a text. However, a more nuanced understanding requires extracting structured sentiment, which involves identifying not just the sentiment polarity but also the holder of the sentiment, the target of the sentiment, and the expression used to convey it. This is often represented as an opinion tuple (target, holder, expression, polarity). The challenge lies in accurately extracting all these interconnected elements, especially when they overlap or are expressed in complex ways. Previous methods often tackled this by breaking it down into separate span extraction and relation prediction tasks, which can lead to error propagation. This paper addresses this by proposing a unified graph parsing approach to directly predict the relationships between sentiment elements.

Methods

The ZHIXIAOBAO system frames structured sentiment analysis as a dependency graph parsing problem. Each sentence is represented as a graph where tokens are nodes and directed edges represent relationships between them.

  • Text Encoder: The system utilizes pre-trained language models like BERT and RoBERTa to generate contextualized embeddings for each token. They found that fine-tuning these models significantly improved performance.
  • Discrete Cross Attention (DCA): To handle the distinct nature of in-span and out-span relationships, they introduce a multi-task learning framework with separate cross-attention mechanisms for each. This allows the model to share underlying text features while learning distinct parameters for different types of relationships.
  • Graph Parsing: A graph parsing algorithm is used to convert the predicted edge scores into opinion tuples. They employ a thresholding mechanism to determine edge existence and use a "head-first" or "head-final" approach to define span heads for parsing.
  • Suffix Masking Trick: To reduce noise from sub-word tokens (suffixes) that offer little discriminative power, they mask these suffixes during edge score computation. This forces the model to rely on more meaningful parts of words for prediction.

Datasets

The system was evaluated on seven datasets across five languages:

  • NoReCFine: Norwegian, multi-domain, large structured sentiment dataset.
  • MultiBCA: Basque, hotel reviews.
  • MultiBEU: Catalan, hotel reviews.
  • OpeNERES: Spanish, polarity-enhanced customer reviews.
  • OpeNEREN: English, polarity-enhanced customer reviews.
  • MPQA: English, news articles.
  • DSUnis: English, online universities and e-commerce reviews.

Key Insights and Ideas

  • Graph Parsing is Superior: Framing structured sentiment analysis as graph parsing, rather than sequential span extraction and relation prediction, is more effective for handling overlapping spans and complex relationships.
  • Fine-tuning Pre-trained Models is Crucial: Fine-tuning large pre-trained language models like RoBERTa significantly boosts performance, especially in low-resource scenarios.
  • DCA for Distinct Relationships: Separating the prediction of in-span and out-span relationships using discrete cross-attention leads to better performance, suggesting these relationship types have different underlying properties.
  • Suffix Masking Reduces Noise: Masking less informative sub-word tokens (suffixes) helps the model focus on more meaningful features, leading to improved accuracy.
  • LSTM Layers are Detrimental: Surprisingly, adding an LSTM layer after the text encoder actually decreased performance, indicating that the direct output of the pre-trained models was sufficient.

Key Results

The ZHIXIAOBAO system achieved outstanding results in SemEval-2022 Task 10:

  • Ranked 1st out of 32 participating teams on the monolingual subtask.
  • Achieved the highest average Sentiment Graph F1 score on 4 datasets.
  • Specifically, they reported SF₁ scores of 72.8 on MultiBCA, 73.9 on MultiBEu, 72.2 on OpeNERES, 76.0 on OpeNEREN, 44.7 on MPQA, and 49.4 on DSUnis when using RoBERTa.

Limitations and Future Work

  • Monolingual Model Availability: For the Norwegian dataset (NoReCFine), a monolingual RoBERTa model was not available at the time of experiments, limiting their ability to achieve the absolute best results on that specific dataset.
  • Limited Training Steps for Pre-training: They attempted to pre-train RoBERTa on a Norwegian corpus but were limited by computational resources and time, resulting in only 2 million training steps. This prevented the model from outperforming the BERT-based approach on that specific dataset.
  • Enriching Contextual Features: While they experimented with incorporating word embeddings, POS tag embeddings, and lemma embeddings, these did not yield superior results compared to directly fine-tuning the pre-trained language model.

Future work could explore more extensive pre-training on specific languages and investigate alternative methods for incorporating richer linguistic features if direct fine-tuning proves insufficient for certain complex cases.

Summary updated

BibTeX
@inproceedings{lin2022zhixiaobao,
  author = {Yangkun Lin and Chen Liang and Jing Xu and Chong Yang and Yongliang Wang},
  title = {{ZHIXIAOBAO at SemEval-2022 Task 10: Apporoaching Structured Sentiment with Graph Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure