UFRGSent at SemEval-2022 Task 10: Structured Sentiment Analysis using a Question Answering Model

Lucas Pessutto, Viviane MoreiraNAACL 2022

TL;DR

This paper introduces UFRGSent, a novel approach to Structured Sentiment Analysis that leverages a Question Answering (QA) model to extract opinion tuples (holder, target, opinion). By framing sentiment extraction as a QA task, they achieve good coverage and a multilingual solution, demonstrating the potential of this paradigm for complex information extraction.

What Problem Does This Paper Address?

The internet is awash with opinions, making it a valuable resource for consumers and businesses alike. However, the sheer volume of unstructured text makes it challenging to automatically process and understand these opinions. Sentiment Analysis aims to tackle this by extracting structured information from text. A specific sub-field, Aspect-Based Sentiment Analysis (ABSA), goes further by identifying and rating specific features (aspects) of an entity being reviewed. Traditionally, ABSA has focused on individual components of an opinion (like aspects or sentiment words) separately. Task 10 of SemEval-2022, "Structured Sentiment Analysis," challenged researchers to identify opinion elements (holder, target, opinion expression, and polarity) together in a structured graph format, treating it as an information extraction problem. This paper addresses the need for a unified approach to extract these interconnected opinion components.

Methods

The UFRGSent system employs a two-phase process. First, a pre-trained multilingual BERT model, fine-tuned for Question Answering, is used to extract candidate opinion tuples. This is achieved through an iterative questioning strategy:

  • Tier 1 Questions: General questions are posed to extract individual components (holder, target, opinion expression).
  • Tier 2 Questions: Based on the extracted components, more specific questions are asked to find the second component of a tuple.
  • Tier 3 Questions: The process continues to find the third component, aiming to form complete (h, t, e) tuples.

The system can also generate "null" tuples if no answer is found, indicating the absence of sentiment. After generating a list of candidate tuples, a ranking and selection mechanism is applied to choose the most representative ones. Finally, a separate, pre-trained Aspect Sentiment Classification model (LCF-BERT) is used to determine the polarity (positive, negative, neutral) for each extracted tuple.

Datasets

The system was evaluated on seven datasets:

  • NoReC: Norwegian professional reviews.
  • MultiBooked_eu & MultiBooked_ca: Hotel reviews in Basque and Catalan.
  • OpeNER_en & OpeNER_es: Hotel reviews in English and Spanish.
  • MPQA: News wire articles in English.
  • Darmstadt_unis: English reviews from online universities.

The authors also mention using a "training data provided" for fine-tuning the QA model, likely specific to the SemEval task.

Key Insights and Ideas

  • QA as a Unified Extraction Framework: The core innovation is framing the complex task of extracting structured sentiment tuples as a series of question-answering problems. This allows for a more holistic extraction of interconnected components.
  • Iterative Questioning for Coverage: The multi-tier questioning approach systematically builds up candidate tuples, leading to good coverage of sentiment expressions in the text.
  • Multilingual by Design: By using a multilingual BERT model, the system is inherently capable of processing text in multiple languages without needing separate models for each.
  • The Trade-off in Candidate Generation: Producing more candidate answers per question can increase the chance of finding the correct tuple but also introduces noise. The authors found that generating too many answers led to a performance drop.

Key Results

The UFRGSent system achieved an average rank of 20th out of 31 participating systems in the SemEval-2022 Task 10 competition. While not top-ranked, the system demonstrated strong coverage, achieving at least 70% coverage in overlap experiments across all datasets. For exact match experiments, coverage exceeded 65% on the Darmstadt_unis and MPQA datasets. The system performed particularly well on the MultiBooked_ca, NoReC, OpeNER_es, and OpeNER_en datasets.

Limitations and Future Work

The authors acknowledge that their candidate ranking and selection methods were simple heuristics and leave their refinement for future work. They are also interested in comparing the performance of their multilingual model against a monolingual version and exploring how other state-of-the-art QA models could further improve sentiment tuple extraction. The paper also notes that the holder information was not used by the Aspect Sentiment Classification model, which could be an area for improvement.

Summary updated

BibTeX
@inproceedings{pessutto2022ufrgsent,
  author = {Lucas Pessutto and Viviane Moreira},
  title = {{UFRGSent at SemEval-2022 Task 10: Structured Sentiment Analysis using a Question Answering Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure