SLPL-Sentiment at SemEval-2022 Task 10: Making Use of Pre-Trained Model’s Attention Values in Structured Sentiment Analysis

Sadrodin BarikbinNAACL 2022

TL;DR

This paper tackles the complex task of Structured Sentiment Analysis (SSA), which goes beyond simple positive/negative sentiment to extract specific sentiment expressions, their targets, and holders, along with their polarities, forming a graph. The authors propose a novel approach that leverages pre-trained transformer models, specifically by utilizing their attention values to predict relationships between sentiment entities, achieving state-of-the-art results on the SemEval-2022 Task 10.

What Problem Does This Paper Address?

Traditional sentiment analysis often focuses on classifying the overall sentiment of a text. However, many real-world applications require a more granular understanding. Structured Sentiment Analysis (SSA) aims to address this by extracting not just the sentiment, but also what is being talked about (target) and who is expressing it (holder), along with the sentiment's polarity. This forms a structured representation, often a graph, which is significantly more informative. The SemEval-2022 Task 10 specifically challenges researchers to extract these "quadruples" (sentiment expression, target expression, holder expression, polarity) from sentences, including both monolingual and cross-lingual scenarios. The authors' work focuses on the monolingual track, aiming to build an efficient and effective model for this complex extraction task.

Methods

The core of the proposed method lies in leveraging pre-trained transformer models, specifically BERT and LaBSE, and making efficient use of their internal attention mechanisms. The system is divided into two main modules:

  • Node Extractor: This module is responsible for identifying the sentiment, target, and holder expressions within a sentence. It uses three separate feed-forward neural networks, each trained with a cross-entropy loss, to predict BIO (Beginning, Inside, Outside) labels for each token corresponding to these entity types. Crucially, it also integrates polarity prediction (negative, neutral, positive) directly into the sequence labeling task by replicating BIO labels for each polarity. This module is designed to be parameter-efficient, with each network having fewer than 50k parameters.
  • Edge Predictor: This module focuses on determining the relationships (edges) between the extracted entities. Instead of training a separate complex model, the authors ingeniously utilize the pre-computed attention values from the pre-trained transformer. They explore several strategies:
  • Base: This approach uses the attention values from a specific head in a specific layer of the pre-trained model. The sum of attention between two entities is used to predict the probability of an edge.
  • +AvgH: This variant aggregates attention values from all heads across all layers, learning a linear combination of these attentions with new learnable weights.
  • +SepH: This setting uses separate attention heads for predicting edges between sentiment-target pairs and sentiment-holder pairs, recognizing that these relationships might have different attentional patterns.
  • +SepH+AvgH: A combination of the previous two, using separate attention aggregations for different pair types.

The overall system architecture is depicted in Figure 1, showing the pre-trained model feeding into the Node Extractor and Edge Predictor, culminating in sequence classification.

Datasets

The authors trained and evaluated their model on seven datasets from the SemEval-2022 Task 10, covering five languages: English, Spanish, Norwegian, Catalan, and Basque. These datasets include:

  • NoReCFine: Norwegian professional reviews.
  • MultiBCA, MultiBEU, Openeren, Openeres: Hotel reviews in Catalan, Basque, English, and Spanish, respectively.
  • DarmstadtUnis: English online university reviews.
  • MPQA: English news articles.

The evaluation metric used for the task is Sentiment Graph F1, which measures the exact match at the graph level, considering overlaps between predicted and gold entities, averaged across entity types.

Key Insights and Ideas

  • Attention as a Relationship Predictor: The most significant insight is the effective use of pre-trained model's attention values for predicting relationships between entities. This avoids the need for complex, separate relation extraction modules and leverages the inherent semantic understanding captured by transformers.
  • Parameter Efficiency: By relying heavily on pre-trained models and carefully designing the Node Extractor, the authors achieve impressive results with a relatively small number of new parameters (around 150k).
  • Joint Prediction of Expressions and Polarities: Integrating polarity prediction directly into the sequence labeling task for expression extraction simplifies the pipeline and potentially improves performance by allowing the model to learn these aspects jointly.
  • Task-Specific Attention Heads: The exploration of separate attention heads for different entity pairs (+SepH) suggests that specific attention mechanisms within transformers might be more attuned to particular types of semantic relationships.

Key Results

The paper reports that their proposed model, "Attentionist," outperforms the baselines on all datasets for the SemEval-2022 Task 10. While specific numerical results are presented in Table 4, the key takeaway is that their approach achieves superior Sentiment Graph F1 scores compared to the GP (graph parsing) and SL-BILSTM (sequence labeling) baselines. The ablation study (Table 3) indicates that the different variants of their attention-based edge prediction contribute to performance, with the base model and the +SepH variant showing promising results.

Limitations and Future Work

  • Pre-trained Model Dependency: The model's performance is heavily reliant on the quality of the pre-trained transformer. While they use state-of-the-art models, their effectiveness can vary across languages and domains.
  • Cross-Lingual Performance: While the task included cross-lingual datasets, the paper primarily focuses on the monolingual track. Further investigation into cross-lingual transfer learning and its effectiveness with their attention-based approach would be valuable.
  • Computational Cost of Attention Aggregation: While parameter-efficient, the +AvgH and +SepH+AvgH settings involve aggregating attention from many heads and layers, which could increase computational cost during inference.
  • Future Work: The authors suggest exploring more sophisticated ways to utilize attention values, potentially incorporating them into the Node Extractor as well. Further research into optimizing the selection of attention heads and layers for specific relation types could also yield improvements.

Summary updated

BibTeX
@inproceedings{barikbin2022slpl,
  author = {Sadrodin Barikbin},
  title = {{SLPL-Sentiment at SemEval-2022 Task 10: Making Use of Pre-Trained Model’s Attention Values in Structured Sentiment Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure