LyS_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing

Iago Alonso-Alonso, David Vilares, Carlos Gómez-Rodríguez

This paper describes the participation of the LyS_ACoruña team in SemEval-2022 Task 10, focusing on Structured Sentiment Analysis (SSA). Instead of building a custom architecture from scratch, the authors demonstrate that SSA can be effectively framed as a semantic dependency parsing problem, allowing them to leverage powerful, "off-the-shelf" parsing tools and pre-trained language models.

TL;DR

The authors treat structured sentiment analysis as a graph-based dependency parsing task, using a bi-affine parser and large language models like XLM-RoBERTa. Their approach achieved competitive rankings (8th and 9th) by showing that general-purpose parsing tools can be successfully repurposed for complex, multi-element sentiment extraction across several languages.

What Problem Does This Paper Address?

Traditional sentiment analysis often stops at classifying a whole sentence as "positive" or "negative." However, real-world opinions are more complex: one sentence might contain multiple opinions from different people about different features.

Structured Sentiment Analysis (SSA) aims to extract these nuances by identifying four-part tuples:

  1. Holder: Who is expressing the opinion?
  2. Target: What is the opinion about?
  3. Sentiment Expression: Which words convey the feeling?
  4. Polarity: Is it positive, negative, or neutral?

As shown in Figure 1, these elements form a graph of relationships. The challenge lies in predicting these structures accurately, especially in "zero-shot" scenarios where no training data exists for a specific target language.

Figure 1: An example of sentiment graph as defined by Barnes et al. (2021). The sentence has a holder (‘I’), two sentiment expressions (‘got’ and ‘at no cost’) and one target (‘an upgrade to Executive suite’)

Methods

The core idea is to view SSA as semantic dependency parsing.

  • The Parser: They used a state-of-the-art bi-affine graph-based parser (Dozat and Manning, 2018) via the supar Python package. This model uses bidirectional LSTMs to create word representations and bi-affine attention to predict edges (relationships) between words.
  • Graph Conversion: To make sentiment data compatible with a dependency parser, they converted the tuples into "head-final" graphs. In this setup, the last token of a sentiment expression acts as a "root," and the last tokens of the holder and target spans are its "dependents."
  • Language Models: They fine-tuned a variety of pre-trained models, including BERT, RoBERTa, and XLM-RoBERTa.
  • Zero-Shot Strategy: For languages without training data, they used word-level translations of available datasets (e.g., translating English to Basque) to create "noisy" but annotated training sets.
  • Training Strategies: They tested three setups: training on single treebanks, merging all available treebanks into one giant dataset, and a hybrid approach that merged training files but kept development files separate for better model selection.

Datasets

The team evaluated their models on 7 treebanks across 5 languages, covering diverse domains like hotel reviews, professional reviews, and news articles.

Table 1: Dataset Statistics

The languages included English, Spanish, Catalan, Basque, and Norwegian. For the zero-shot task, Basque, Catalan, and Spanish were the target languages.

Key Insights and Ideas

  • XLM-RoBERTa is a Powerhouse: Surprisingly, the cross-lingual XLM-RoBERTa model often outperformed language-specific models even in monolingual tasks. Its broad pre-training seems to capture sentiment nuances that narrower models miss.
  • Data Merging Helps: Merging training data from different treebanks (even across different domains) generally improved performance. This suggests that the underlying "grammar" of sentiment is consistent enough that models benefit from more diverse examples.
  • Direct Transfer > Noisy Translation: In their post-evaluation analysis, the authors found that simply training a model on English data and applying it directly to other languages (using XLM-RoBERTa) worked better than using the noisy, word-level translated datasets they initially created.
  • Domain Difficulty: Performance varied wildly by domain. Models excelled at hotel reviews (OpeNER/MultiBooked) but struggled with news articles (MPQA), which often contain more complex, indirect opinions.

Key Results

The team ranked 8th in the monolingual setup and 9th in the cross-lingual setup officially.

  • Monolingual Performance: On the English OpeNER dataset, their best model achieved an F1 score of 0.714. However, on the more difficult MPQA news dataset, the F1 dropped to 0.374.
  • Cross-Lingual (Zero-Shot): Their initial translation-based approach reached F1 scores around 0.50–0.57. Their post-evaluation "direct transfer" baseline (training on English and testing on others) significantly boosted these scores, reaching up to 0.670 for Spanish.

Limitations and Future Work

The authors noted that their word-level translation approach for zero-shot learning produced "unlikely-grammatical" data, which likely limited the parser's performance. Furthermore, the significant performance gap between domains (hotels vs. news) suggests that current models still struggle with the subtle ways sentiment is expressed in formal or objective-sounding text.

Future work could involve more sophisticated data augmentation or exploring how to better bridge the gap between disparate domains like social media reviews and journalistic prose.

Summary updated

Abstract

This paper addressed the problem of structured sentiment analysis using a bi-affine semantic dependency parser, large pre-trained language models, and publicly available translation models. For the monolingual setup, we considered: (i) training on a single treebank, and (ii) relaxing the setup by training on treebanks coming from different languages that can be adequately processed by cross-lingual language models. For the zero-shot setup and a given target treebank, we relied on: (i) a word-level translation of available treebanks in other languages to get noisy, unlikely-grammatical, but annotated data (we release as much of it as licenses allow), and (ii) merging those translated treebanks to obtain training data. In the post-evaluation phase, we also trained cross-lingual models that simply merged all the English treebanks and did not use word-level translations, and yet obtained better results. According to the official results, we ranked 8th and 9th in the monolingual and cross-lingual setups.
BibTeX
@inproceedings{alonsoalonso2022lys,
  author = {Iago Alonso-Alonso and David Vilares and Carlos Gómez-Rodríguez},
  title = {{LyS\_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure