TL;DR
This paper introduces a novel "coarse-to-fine grained Image-Target Matching network" (ITM) to improve Targeted Multimodal Sentiment Classification (TMSC). By explicitly modeling both the relevance between an image and a target entity, and the alignment of visual objects within the image to that target, ITM significantly outperforms existing methods. This approach addresses the noise introduced by irrelevant images and the ambiguity in fine-grained visual cues, leading to more accurate sentiment predictions.
What Problem Does This Paper Address?
Targeted Multimodal Sentiment Classification (TMSC) aims to determine the sentiment expressed towards specific entities (targets) within a piece of content that includes both text and an image. For example, in a tweet about a movie premiere, TMSC would identify the sentiment towards "the movie" and "the red carpet." Existing methods often struggle because they don't adequately consider how the image relates to the specific target. A significant portion of targets in multimodal datasets are not visually represented in the accompanying image, leading to "visual noise" that can mislead sentiment analysis. Furthermore, even when an image is relevant, it contains many visual objects, and it's crucial to identify which specific object(s) are most relevant to the target to extract the most informative visual cues. This paper tackles these two key challenges: the relevance of the image to the target and the fine-grained alignment of visual objects to the target.
Methods
The proposed ITM network is a multi-task learning architecture designed to address the identified problems. It comprises four main modules:
- Feature Extraction: This module first obtains a contextualized representation of the target entity from the text using a pre-trained language model (RoBERTa). For the image, it uses Faster R-CNN to detect object proposals and then employs a Transformer to generate object-level image representations.
- Coarse-Grained Matching (Image-Target Relevance): This module uses a Cross-Modal Transformer to model the interaction between the image and target representations. It then predicts a relevance score indicating whether the image is related to the target. This score is used to create a "visual filter" that down-weights or removes features from irrelevant images, mitigating visual noise.
- Fine-Grained Matching (Object-Target Alignment): Building on the filtered image representations, this module uses another Cross-Modal Transformer to learn an attention distribution over the detected object proposals. This distribution indicates which visual objects are most aligned with the target. This alignment is guided by a ground-truth alignment distribution derived from bounding box IoU scores.
- Multimodal Fusion: Finally, the target representation is fused with the filtered, object-aligned image representation. This fused representation is then fed into a Transformer for multimodal fusion, and the output is used to predict the final sentiment polarity for the target.
The model is trained using a combined objective function that includes losses for image-target relevance classification, object-target alignment (measured by KL divergence), and the main TMSC task (sentiment classification).
Datasets
The paper utilizes two types of datasets:
- Image-Target Matching Dataset: This is a newly constructed dataset created by manually annotating a subset of the TWITTER-17 dataset. It includes annotations for image-target relevance (Related/Unrelated) and bounding box alignments for visual objects. This dataset is crucial for training the auxiliary tasks.
- Benchmark TMSC Datasets: The model is evaluated on two standard benchmark datasets for TMSC: TWITTER-15 and TWITTER-17.
Key Insights and Ideas
- Image-Target Relevance is Crucial: A significant percentage of targets in multimodal datasets are not visually present in the images. Explicitly modeling this relevance allows the model to filter out noisy, irrelevant images, which is essential for accurate sentiment prediction.
- Fine-Grained Alignment Matters: Even when an image is relevant, identifying the specific visual object(s) most aligned with the target entity provides more discriminative information than using the entire image.
- Coarse-to-Fine Strategy is Effective: The proposed two-stage matching process (relevance filtering followed by object alignment) effectively leverages visual information, leading to substantial performance gains.
- Interpretable Visualizations: The paper provides visualizations that demonstrate how the model learns image-target relevance and object-target alignment, offering insights into its decision-making process.
Key Results
The ITM model achieves state-of-the-art results on both the TWITTER-15 and TWITTER-17 benchmark datasets for TMSC, outperforming various text-only and multimodal methods. For instance, on TWITTER-17, ITM achieves an accuracy of 78.27% and an F1 score of 71.97%, surpassing strong baselines like TomRoBERTa+Aux-Tasks. Ablation studies confirm that both the image-target relevance and object-target alignment components are indispensable, with their removal leading to significant performance drops. Case studies illustrate how ITM correctly handles cases where other models fail due to irrelevant images or poor object alignment.
Limitations and Future Work
- Limitations: The paper acknowledges that the performance of the object-target alignment module is dependent on the quality of the object detection method (Faster R-CNN in this case). Small or occluded objects might still pose challenges. The reliance on pre-trained models like RoBERTa means the model inherits their limitations.
- Future Work: The authors suggest exploring more advanced object detection techniques and investigating how to better leverage the contextual information within the target itself for alignment. Further research into more sophisticated multimodal fusion strategies could also be beneficial. The interpretability of the model could be further enhanced by exploring more detailed visualization techniques.
Summary updated