Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection

Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, Guodong ZhouEMNLP 2021 · OpenReview

Abstract

Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, Guodong Zhou. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

Aspect-Level Sentiment Analysis (ALSA) involves identifying specific aspect terms in text and determining the sentiment expressed toward them. In the context of social media, where posts are inherently multi-modal, researchers have traditionally split this into two independent tasks: Multi-modal Aspect Term Extraction (MATE) and Multi-modal Aspect Sentiment Classification (MASC).

However, this decoupled approach suffers from two primary issues. First, it ignores the innate connection between identifying an object and judging it. Second, it is susceptible to "visual noise." Not every image accompanying a tweet is relevant to the sentiment of the text. For instance, a sports image might be used metaphorically in a political tweet, potentially misleading a model that forces a connection between the visual and textual sentiment.

Examples of visual noise in multi-modal sentiment analysis where images may or may not be relevant to the text.

Methods

To address these challenges, the authors propose Joint Multi-modal Learning (JML), the first framework to jointly extract aspect terms and predict sentiment polarities in a multi-modal setting.

The JML model architecture featuring auxiliary cross-modal relation detection and task-specific visual guiding.

The JML architecture introduces three key innovations:

  1. Auxiliary Cross-modal Relation Detection: The model includes a "gatekeeper" module trained on the Text-Image Relation (TRC) dataset. It determines if an image adds meaningful information to the text. If the relationship is weak, the model dampens the visual signal to avoid distraction.
  2. Hierarchical Joint Framework: Instead of using a single "collapsed" tag for both aspect and sentiment, JML uses a hierarchical structure. It first identifies candidate aspect spans (start and end positions) and then feeds these specific spans into a sentiment classifier.
  3. Task-Specific Visual Guiding: The model recognizes that MATE and MASC require different levels of visual granularity. Extraction (MATE) might only need coarse-grained visual cues (e.g., identifying a "person"), whereas Sentiment (MASC) requires fine-grained details like facial expressions.

The system utilizes BERT for textual representations and ResNet for image features, employing cross-modal attention to fuse the modalities effectively.

Key Results

The JML model was evaluated against several state-of-the-art baselines on the Twitter-2015 and Twitter-2017 benchmarks.

  • Overall Performance: JML achieved a Micro F1 score of 64.1% on Twitter-2015 and 66.0% on Twitter-2017, significantly outperforming both text-only joint models and previous multi-modal approaches.
  • Ablation Studies: Removing the Relation Detection module led to a performance drop, confirming that filtering irrelevant visual information is critical.
  • Joint Learning Benefits: JML set new records for both MATE and MASC sub-tasks individually, demonstrating that joint learning allows the model to share features that benefit both extraction and classification.

Conclusion

JML demonstrates that explicitly modeling the relationship between text and images is vital for multi-modal sentiment analysis. By acting as a gatekeeper for visual information and utilizing a hierarchical joint framework, JML effectively mitigates visual noise and leverages the synergy between aspect extraction and sentiment classification. Future work may explore extending this approach to more complex scenarios like multi-modal dialogue systems or utilizing self-supervised learning to reduce reliance on annotated relation datasets.

Summary updated

BibTeX
@inproceedings{ju2021joint,
  author = {Xincheng Ju and Dong Zhang and Rong Xiao and Junhui Li and Shoushan Li and Min Zhang and Guodong Zhou},
  title = {{Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=AMmkMCItGL9},
}
Figure