Integrated Multi-stage Contextual Attention Network for Text-Image Matching

Jiande Sun, Yi Shao, Yawen Chen, Yang Zhang, Tianlin Zhang, Xuan Zhang, Ye Jiang, Jing LiMediaEval 2023 · OpenReview

TL;DR

This paper introduces the Integrated Multi-stage Contextual Attention Network (IMCAN) for text-image matching, a crucial task for cross-modal retrieval. IMCAN leverages multi-stage Transformer encoders and a novel Multi-modal Contextual Attention Network (MCAN) to effectively fuse textual and visual features, achieving improved alignment and semantic understanding between text and images.

What Problem Does This Paper Address?

The paper tackles the challenge of text-image matching, which is fundamental for tasks like cross-modal retrieval where the goal is to find relevant images for a given text query, or vice-versa. While significant progress has been made, accurately aligning and fusing information from text and images remains difficult due to their inherent differences in representation and potential semantic divergences. The NewsImages task within MediaEval 2023 highlights the ongoing need for robust models that can effectively bridge this gap.

Methods

The core of the proposed IMCAN model lies in its multi-stage architecture. It begins by extracting textual features using BERT and visual features using Vision Transformer (ViT). These uni-modal features are then processed through multiple stages of Transformer encoders. Crucially, a Multi-modal Contextual Attention Network (MCAN) is integrated. This MCAN comprises two contextual attention blocks that operate on the features from different Transformer stages, allowing for a deeper fusion and alignment of textual and visual information. This multi-stage approach aims to capture increasingly complex semantic relationships. The model is trained end-to-end, optimizing for precise feature matching, and uses cosine similarity to measure the distance between text and image representations.

Datasets

The paper evaluates IMCAN on the following datasets:

  • GDELT-1 (English)
  • GDELT-2 (English)
  • RT (German, machine-translated to English for standard CLIP evaluation)

Key Insights and Ideas

  • Multi-stage processing is beneficial: The paper demonstrates that increasing the number of Transformer encoder stages (up to 2) leads to improved performance in text-image matching. This suggests that progressively refining and fusing features at different levels of abstraction is key.
  • Overfitting risk with too many stages: Beyond two stages, performance begins to degrade, indicating a risk of overfitting and memorizing noise when the model becomes too complex for the available data.
  • Loss function impact: While both cosine similarity loss and triplet loss were explored, the paper notes that cosine similarity loss yields higher recall rates on smaller test subsets, suggesting it might be more robust in scenarios with limited data or specific sample distributions. Triplet loss, however, shows potential for finding good matches in larger datasets.
  • CLIP's strong baseline: The results highlight CLIP as a strong baseline, especially on English datasets, underscoring the effectiveness of large-scale pre-training. However, IMCAN aims to provide a more specialized and potentially more interpretable solution for specific cross-modal retrieval tasks.

Key Results

  • Comparative Performance: On the GDELT-1 and GDELT-2 datasets, IMCAN's performance (MRR@100, Recall@5, Recall@10, Recall@50, Recall@100) is presented in Table 2. While not directly compared head-to-head with CLIP in the provided tables for the same experimental setup, the paper notes that CLIP fine-tuned on these datasets shows excellent retrieval performance.
  • Ablation Study: The ablation study (Table 3) on GDELT-1 shows that a 2-stage IMCAN model achieved the best performance in terms of MRR and recall rates when trained with cosine similarity loss. Performance improved from 0 to 2 stages but declined from 3 to 4 stages.
  • Loss Function Comparison: For GDELT-1 and GDELT-2, the cosine similarity loss generally yielded slightly higher MRR and recall scores compared to triplet loss, especially on smaller test subsets.

Limitations and Future Work

  • Performance Gap with CLIP: The authors acknowledge that IMCAN's performance on English datasets lags significantly behind CLIP. They attribute this partly to differences in computing resources, training data volume, and batch size, but also investigate the model's structural rationality.
  • Optimal Number of Stages: The study suggests that 2 stages are optimal for IMCAN, with more stages leading to overfitting. This implies a need for careful architectural design and hyperparameter tuning.
  • Future Work: The paper doesn't explicitly detail future work, but the findings suggest avenues such as exploring more advanced attention mechanisms, investigating different fusion strategies, and further optimizing the training process to close the performance gap with state-of-the-art models like CLIP. The potential for multilingual extensions of IMCAN is also an implied area for future exploration, given the RT dataset's German origin.

Summary updated

BibTeX
@inproceedings{sun2023integrated,
  author = {Jiande Sun and Yi Shao and Yawen Chen and Yang Zhang and Tianlin Zhang and Xuan Zhang and Ye Jiang and Jing Li},
  title = {{Integrated Multi-stage Contextual Attention Network for Text-Image Matching}},
  booktitle = {MediaEval},
  year = {2023},
  url = {https://openreview.net/forum?id=2aDXIkKaXL},
}
Figure