Graph Pattern Loss Based Diversified Attention Network For Cross-Modal Retrieval

Xueying Chen, Rong Zhang, Yibing Zhan

TL;DR

This paper introduces GPLDAN, an unsupervised cross-modal retrieval network that uses a "diversified attention" mechanism to create multiple representations of data and a novel "graph pattern loss" to capture complex relationships between these representations. This approach significantly improves retrieval accuracy by better understanding the correlations within and across different data modalities.

What Problem Does This Paper Address?

Cross-modal retrieval, the task of finding relevant information across different data types (like images and text), is crucial for handling the vast amount of multimedia data available today. A major challenge is the inherent "heterogeneity" of these modalities – images and text are represented and distributed very differently, making it hard to directly compare them. While supervised methods use labels to bridge this gap, obtaining such labels is often expensive and time-consuming. Unsupervised methods aim to learn these relationships without labels, but existing approaches struggle to fully capture the intricate correlations between different representations of the same or different objects. This paper tackles the problem of how to effectively learn these complex, multi-faceted correlations in an unsupervised manner to achieve robust cross-modal retrieval.

Methods

The proposed Graph Pattern Loss based Diversified Attention Network (GPLDAN) has three main components:

  1. Diversified Attention Feature Projector: This module aims to extract richer representations by considering interactions within and between modalities. It first applies a denoising operation to input features and then uses weight sharing to project image and text features into a common space. Crucially, it generates k multiple representations for each instance by employing both self-attention (to capture intra-modality relationships) and co-attention (to capture inter-modality synergistic relationships).
  2. Graph Pattern Loss: This novel loss function is designed to explore the complex correlations among the multiple representations generated by the diversified attention. It considers all possible distances between these representations. It comprises three parts:
  3. Pairwise Distance Loss: Encourages representations of the same object from different modalities to be similar.
  4. Unpaired Distance Preserving Loss: Ensures that representations of different objects are dissimilar, both within and across modalities.
  5. Mutual Distance Preserving Loss: Further emphasizes the relationship between cross-modal instances by aligning their distance distributions.
  6. Modality Classifier: An auxiliary classifier is added to explicitly predict the modality of features before fusion. This helps guide the feature projector to learn more discriminative representations and enhances the stability of the graph pattern loss.

Datasets

The authors evaluate GPLDAN on four publicly available datasets:

  • Wikipedia: A dataset for image-text retrieval.
  • Pascal Sentence: Another dataset for image-text retrieval.
  • NUS-WIDE-10K: A large-scale dataset for image-text retrieval.
  • XMedia: A dataset for cross-modal retrieval.

For image features, they use 4,096d vectors from the fc7 layer of VGG-19 (for Wikipedia, Pascal Sentence, NUS-WIDE-10K) or AlexNet (for XMedia). For text features, they use 1,024d vectors from BERT (for the first three datasets) or 3,000d bag-of-words vectors (for XMedia).

Key Insights and Ideas

  • Diversified Representations are Key: Generating multiple representations of an instance using self- and co-attention allows the network to capture a richer set of relationships than a single representation.
  • Graph-based Correlation Modeling: The graph pattern loss provides a comprehensive way to model relationships by considering all pairwise and unpaired distances, going beyond simple similarity metrics. This allows for a more nuanced understanding of how different representations relate to each other.
  • Unpaired Distances Matter: Explicitly enforcing that representations of different objects are far apart (unpaired distance preserving loss) is crucial for good performance, highlighting the importance of negative sampling in unsupervised learning.
  • Modality Awareness Aids Learning: The modality classifier, by forcing the network to explicitly recognize feature modalities before fusion, helps improve the discriminative power of the learned representations.

Key Results

GPLDAN demonstrates strong performance across the evaluated datasets. It outperforms most unsupervised cross-modal retrieval methods and is competitive with semi-supervised and supervised approaches. For instance, on the Pascal Sentence dataset, GPLDAN achieves an average MAP@50 of 0.469, outperforming other unsupervised methods like CDPAE (0.409) and DCCA (0.360). In particular, for the Text-to-Image retrieval tasks on Pascal Sentence and NUS-WIDE-10K, GPLDAN outperforms all other methods (supervised included). The paper also shows that the unpaired distance preserving loss significantly boosts performance (24.8% improvement over baseline), and the diversified attention further enhances results.

Limitations and Future Work

  • Parameter Tuning: The performance of the graph pattern loss is sensitive to the parameters $\alpha$ and $\beta$, which require careful tuning per dataset.
  • Computational Cost: While the graph pattern loss is applied within mini-batches, the diversified attention mechanism and the consideration of multiple distances might increase computational complexity compared to simpler methods.

The authors suggest that their approach has good generation ability and can be applied to various pre-trained image and text features, indicating potential for broader applicability. Future work could explore more efficient ways to model the graph patterns or investigate adaptive parameter tuning strategies.

Summary updated

Abstract

Cross-modal retrieval aims to enable flexible retrieval experience by combining multimedia data such as image, video, text, and audio. One core of unsupervised approaches is to dig the correlations among different object representations to complete satisfied retrieval performance without requiring expensive labels. In this paper, we propose a Graph Pattern Loss based Diversified Attention Network (GPLDAN) for unsupervised cross-modal retrieval to deeply analyze correlations among representations. First, we propose a diversified attention feature projector by considering the interaction between different representations to generate multiple representations of an instance. Then, we design a novel graph pattern loss to explore the correlations among different representations, in this graph all possible distances between different representations are considered. In addition, a modality classifier is added to explicitly declare the corresponding modalities of features before fusion and guide the network to enhance discrimination ability. We test GPLDAN on four public datasets. Compared with the state-of-the-art cross-modal retrieval methods, the experimental results demonstrate the performance and competitiveness of GPLDAN.
BibTeX
@inproceedings{chen2020graph,
  author = {Xueying Chen and Rong Zhang and Yibing Zhan},
  title = {{Graph Pattern Loss Based Diversified Attention Network For Cross-Modal Retrieval}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=30Q28L8KKZ},
}
Figure