AMEX AI Labs at SemEval-2022 Task 10: Contextualized fine-tuning of BERT for Structured Sentiment Analysis

Pratyush Sarangi, Shamika Ganesan, Piyush Arora, Salil JoshiNAACL 2022

TL;DR

This paper introduces CBERT, a novel BERT-based model for Structured Sentiment Analysis (SSA) that effectively combines contextualized BERT embeddings with syntactic and pre-trained embeddings. CBERT achieves competitive results on the SemEval-2022 Task 10, outperforming previous BiLSTM-based approaches and demonstrating the value of integrating diverse linguistic information for fine-grained sentiment extraction.

What Problem Does This Paper Address?

Structured Sentiment Analysis (SSA) goes beyond traditional sentiment analysis by aiming to extract not just the overall sentiment of a text, but also the specific opinion elements: the source of the opinion, the target of the opinion, and the polar expression itself. This task is challenging because it requires identifying these elements and understanding their relationships within a sentence. Existing approaches often use separate models for each element, which can lead to a loss of information. The authors aim to develop a more robust and generalized approach that can effectively capture these complex relationships across different domains and languages.

Methods

The core of the proposed system, CBERT, is a BERT-based encoder. What makes CBERT novel is its approach to integrating different types of embeddings:

  • BERT Embeddings: The system leverages a pre-trained BERT base multilingual cased model to capture contextual information from the input text.
  • Syntactic and Pre-trained Embeddings: CBERT also incorporates syntactic embeddings from Stanza (POS tags, lemmas, dependency parses) and ELMo embeddings.
  • Novel Concatenation: These diverse embeddings are combined using a unique concatenation mechanism. Specifically, the BERT output is concatenated with independent BiLSTM blocks that process the head and dependent contexts of the input. This allows the model to learn from both the contextualized BERT representation and the structured linguistic information.
  • Attention Layer: An attention layer is used to predict the relationships (arcs) between the opinion elements, transforming the concatenated embeddings into a tensor that captures potential connections between sources, targets, and polar expressions.
  • Weighted Loss Function: To address the imbalance in label frequencies (particularly the "None" label), a weighted cross-entropy loss function is employed, with weights empirically determined to optimize performance.

The paper also contrasts CBERT with a simpler VBERT model, which uses only BERT embeddings without the additional BiLSTM blocks for head and dependent contexts.

Datasets

The authors evaluated their system on seven datasets for the monolingual SSA task at SemEval-2022. These datasets cover five languages and include:

  • MPQA
  • NoRec
  • Multibook (ca and eu variants)
  • Opener (en and es variants)
  • Darmstadt

The evaluation metric used is Sentiment-F1 (SF1) score, which requires an exact match for all opinion term elements (source, target, polar expression) to count as a true positive.

Key Insights and Ideas

  • The Power of Combined Embeddings: The most significant insight is that combining contextualized BERT embeddings with syntactic and pre-trained embeddings (via the novel concatenation and BiLSTM approach) leads to improved performance in SSA. This suggests that structured linguistic information complements the rich contextual understanding provided by BERT.
  • Addressing Label Imbalance is Crucial: The authors found that the frequency of "None" labels was significantly higher than other labels, necessitating a weighted loss function to prevent the model from being biased towards predicting "None."
  • Boundary Detection Challenges: The qualitative analysis highlights that identifying precise boundaries for opinion elements can be tricky, especially when dealing with punctuation, stopwords, or adjectives that might be part of the sentiment expression but not explicitly annotated.
  • Tuple Formation Nuances: The system's tendency to form single tuples for sentences with multiple polar expressions for a single target is identified as an area for improvement, suggesting that more granular tuple formation might be needed.
  • Domain-Specific Annotation Gaps: The paper points out that some datasets might have annotations that are specific to their domain (e.g., restaurant reviews), leading to potential issues when the model encounters texts outside that domain but still containing opinion elements.

Key Results

  • CBERT achieved an average rank of 14 out of 32 systems on the SemEval-2022 Task 10.
  • The proposed CBERT model performed relatively better than the VBERT model (which uses only BERT embeddings) and BiLSTM-based approaches, showing a 9% improvement over VBERT.
  • While CBERT performed well on most datasets, it did not perform as strongly on the MPQA and Darmstadt_unis datasets, which the authors attribute to a high number of neutral labels not being explicitly handled.

Limitations and Future Work

  • Neutral Labels: The authors acknowledge that the current system does not explicitly handle neutral labels, which may have impacted performance on datasets with a high proportion of such labels.
  • Overlapping Spans: The post-processing step aims to remove overlapping source or target spans from polar expressions, but this indicates that such overlaps can occur and require specific handling.
  • Future Directions: The authors suggest exploring 2D Conditional Random Fields (CRFs) for improved span detection, investigating different joint learning architectures on top of CBERT's encoding (like SDRN), and exploring approaches to fine-tune models for specific tasks.

Summary updated

BibTeX
@inproceedings{sarangi2022amex,
  author = {Pratyush Sarangi and Shamika Ganesan and Piyush Arora and Salil Joshi},
  title = {{AMEX AI Labs at SemEval-2022 Task 10: Contextualized fine-tuning of BERT for Structured Sentiment Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure