SenPoi at SemEval-2022 Task 10: Point me to your Opinion, SenPoi

Jan Pfister, Sebastian Wankerl, Andreas HothoNAACL 2022

TL;DR

This paper tackles the complex task of extracting structured sentiment tuples (opinion holder, target, sentiment expression, and polarity) from text, going beyond simple sentence-level sentiment. They adapt a pointer network approach (BARTABSA) with a novel, flexible output grammar and introduce an "Output Length Regularizer" to significantly improve performance, especially for multilingual and cross-lingual settings.

What Problem Does This Paper Address?

Traditional sentiment analysis often boils down to classifying a whole sentence or document as positive, negative, or neutral. However, real-world opinions are far more nuanced. A single sentence might express positive sentiment towards a restaurant's food but negative sentiment towards its service. This is where Structured Sentiment Analysis comes in, aiming to extract specific opinion tuples that capture these finer-grained details. The challenge is that these tuples can be complex: entities (like the target or holder) might be missing, spread across multiple words, or even absent entirely. Furthermore, doing this across multiple languages, especially without direct training data for the target language (zero-shot cross-lingual), adds another layer of difficulty. The authors specifically address SemEval-2022 Task 10, which focuses on extracting these complex, arbitrary opinion tuples.

Methods

The core of their approach builds upon BARTABSA, a sequence-to-sequence model that uses a pointer network. This means instead of generating words from a vocabulary, the model "points" to specific tokens in the input text to construct the output.

Here's what's new:

  • Flexible Output Grammar: They introduce a more expressive grammar to represent the structured sentiment tuples. This grammar allows for:
  • Optional Entities: If a holder or target isn't present, it can be omitted without breaking the structure.
  • Split Entities: An entity (like a target) can be composed of multiple, non-contiguous parts of the text.
  • Sentiment Source/Holder: Explicitly models who is expressing the sentiment.
  • Multilingual Capability: They swap the English-centric BART model for a multilingual mT5 model, enabling them to handle languages beyond English.
  • Output Length Regularizer (OLR): A key innovation. They noticed models often overfit to common sequence lengths, particularly the End-Of-Sequence (EOS) token placement. A misplaced EOS token can lead to incomplete or entirely missed sentiment tuples. The OLR adds a penalty to the loss function when the EOS token is predicted too early, encouraging more accurate sequence termination and thus more complete tuple extraction.

Datasets

The paper evaluates their approach on seven datasets across five languages: English, Spanish, Catalan, Basque, and Norwegian. These datasets are structured to include sentiment tuples with varying complexity, including missing and split entities. The challenge also included a cross-lingual setting where models were trained on one set of languages and tested on others, including a zero-shot scenario.

Key Insights and Ideas

  • Grammar is King for Complexity: The success in handling complex sentiment tuples hinges on a flexible output grammar. Simply adapting existing models without accounting for optionality, split entities, and holders would fail.
  • The EOS Token is a Minefield: Overfitting to the EOS token's position is a significant problem, especially for models like mT5. A seemingly small error in EOS placement can drastically impact the extracted sentiment graph.
  • Multilingual Models Need Help: While mT5 offers multilingual capabilities, it's prone to training instabilities and performance drops compared to BART, particularly in complex tasks like structured sentiment analysis. The OLR helps mitigate some of these issues.
  • Zero-Shot is Tough, But Possible: The paper demonstrates that pointer-based models can generalize to unseen languages and domains, even if performance takes a hit.

Key Results

  • The proposed approach, especially with the Output Length Regularizer, significantly improved performance, with gains of up to 70% in some settings.
  • The OLR consistently improved sentiment F1 scores across various datasets and models, particularly benefiting the mT5 model.
  • In the monolingual subtask, their approach achieved a competitive placement (16th out of 31 teams).
  • While mT5 showed training instabilities and lower performance compared to BART, the OLR helped stabilize its training and improve results.
  • In the cross-lingual setting, despite a significant performance drop (over 40% on average), the model still demonstrated the ability to generalize zero-shot.

Limitations and Future Work

  • mT5 Instabilities: The paper highlights that mT5, despite its multilingual nature, suffers from training instabilities and a performance hit compared to BART in this task. This is attributed to pre-training differences and the inherent difficulty of the task.
  • Cross-lingual Performance Drop: While zero-shot cross-lingual generalization is demonstrated, there's a notable performance decrease. Future work could explore more advanced cross-lingual transfer techniques.
  • Dataset Specific Sampling: The authors found that optimal dataset sampling strategies varied between BART and mT5, and even across different datasets for mT5. Further research into robust sampling methods could be beneficial.
  • Crossdomain Generalization: While the model shows some crossdomain generalization, performance drops significantly when domains are dissimilar (e.g., political opinions vs. hotel reviews). Investigating domain adaptation techniques could be a future direction.

Summary updated

BibTeX
@inproceedings{pfister2022senpoi,
  author = {Jan Pfister and Sebastian Wankerl and Andreas Hotho},
  title = {{SenPoi at SemEval-2022 Task 10: Point me to your Opinion, SenPoi}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure