Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysis

Hao Yang, Zhenyu Zhang, Yanyan Zhao, Bing QinarXiv 2024 · DOI · arXiv

Introduction: The Noise in Multimodal Sentiment

Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to determine the sentiment—positive, negative, or neutral—toward a specific entity (an "aspect") mentioned in a social media post containing both text and imagery. While powerful, current MABSA models often struggle with the inherent messiness of real-world data. Social media images are frequently blurry, low-resolution, or entirely irrelevant to the accompanying text.

Previous approaches typically treated all data points with equal weight. This is problematic because low-quality or "uncertain" samples introduce noise, causing models to learn incorrect biases rather than meaningful cross-modal relationships. Researchers from the Harbin Institute of Technology argue that a model must be "uncertainty-aware," distinguishing between high-quality challenging samples and low-quality noise.

Categorization of samples based on quality and difficulty.

The UA-MABSA Framework

To address these challenges, the authors propose UA-MABSA (Uncertainty-Aware MABSA). This framework introduces a novel quality assessment strategy that dynamically weights the loss function during training, prioritizing informative samples.

The architecture consists of three core components:

  1. Image Quality Assessment: The model calculates objective scores for brightness, contrast, sharpness, and resolution using OpenCV. Additionally, it employs PaddleOCR to detect text within images; a high density of text often indicates a "noisy" image for visual feature extraction.
  2. Correlation Assessment: Utilizing the CLIP (Contrastive Language-Image Pre-training) model, the system evaluates relevance at two levels:
  3. Coarse-grained: The alignment between the image and the overall text.
  4. Fine-grained: The specific relevance of the image to the aspect being analyzed.
  5. Weighted Loss Mechanism: The model uses a "Caption Transformer" to generate descriptive text from images, which is then combined with the original post and aspect for processing by a pre-trained language model (e.g., BERT). The final loss for each sample is adjusted by a weight derived from the quality and correlation scores, forcing the model to focus on high-quality, relevant data.
The UA-MABSA architecture incorporating uncertainty indicators.

Key Insights: Hard Samples vs. Noise

A central contribution of this work is the formalization of data uncertainty. The authors distinguish between "low-quality" (noisy) samples and "hard" (complex but high-quality) samples. Their weighting strategy encourages the model to exert more effort on samples where sentiment requires deep cross-modal reasoning, rather than simply relying on easy textual cues or being misled by irrelevant visuals.

Interestingly, the study found that while direct OCR text integration provided marginal gains, using OCR text length as a quality indicator was highly effective for filtering noise.

Experimental Results

UA-MABSA was evaluated on the standard Twitter-2015 and Twitter-2017 benchmark datasets, achieving state-of-the-art (SOTA) performance:

  • Twitter-2015: The model achieved an Accuracy of 78.88% and a Macro-F1 of 74.49%, outperforming established models like CapBERT and FITE.
  • Twitter-2017: The approach remained highly competitive (71.85% Accuracy), demonstrating robustness even when the proportion of low-quality data varies.

Ablation studies confirmed that removing any of the three pillars—Image Quality, Image-Text Relevance, or Aspect-Image Relevance—led to significant performance degradation, proving that a multi-dimensional view of uncertainty is essential.

Future Directions

While UA-MABSA sets a new benchmark, the authors note that current manual thresholds for quality scores could be improved through more adaptive, self-tuning assessment methods. Future work will also look to extend this uncertainty-aware logic to aspect extraction and joint multimodal tasks, further refining how AI interprets the nuanced and often noisy world of social media.

Summary updated

Abstract

As a fine-grained task, multimodal aspect-based sentiment analysis (MABSA) mainly focuses on identifying aspect-level sentiment information in the text-image pair. However, we observe that it is difficult to recognize the sentiment of aspects in low-quality samples, such as those with low-resolution images that tend to contain noise. And in the real world, the quality of data usually varies for different samples, such noise is called data uncertainty. But previous works for the MABSA task treat different quality samples with the same importance and ignored the influence of data uncertainty. In this paper, we propose a novel data uncertainty-aware multimodal aspect-based sentiment analysis approach, UA-MABSA, which weighted the loss of different samples by the data quality and difficulty. UA-MABSA adopts a novel quality assessment strategy that takes into account both the image quality and the aspect-based cross-modal relevance, thus enabling the model to pay more attention to high-quality and challenging samples. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance on the Twitter-2015 dataset. Further analysis demonstrates the effectiveness of the quality assessment strategy.
BibTeX
@misc{yang2024data,
  author = {Hao Yang and Zhenyu Zhang and Yanyan Zhao and Bing Qin},
  title = {{Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysis}},
  year = {2024},
  eprint = {2412.01249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01249},
}
Figure