Mining on Heterogeneous Manifolds for Zero-Shot Cross-Modal Image Retrieval

Fan Yang, Zheng Wang, Jing Xiao, Shin'ichi Satoh

Introduction

Standard cross-modal retrieval models often struggle with "zero-shot" scenarios, where the categories of objects in the test set were never seen during training. The primary challenge lies in mapping different modalities (e.g., visible RGB and thermal infrared) into a single, uniform feature space. While this approach works for training data, it frequently distorts the natural structure—or manifolds—of unseen classes, leading to fragmented feature distributions and poor retrieval performance.

The Problem: Manifold Distortion

In cross-modal retrieval, the goal is to find a match for a query in one modality within a gallery of another. Current "two-stream" models attempt to reconcile conflicting relationships between modalities during training. For example, a lemon might be closer to a mango in color but closer to a lime in shape. Forcing these relationships into one space can cause the feature manifold of an unseen class to become cluttered and disconnected.

t-SNE visualizations of feature distributions across different modality mappings, illustrating how joint spaces can distort manifolds of unseen classes.

As shown in the t-SNE visualizations above, joint models can "shatter" clusters of unseen data into isolated blobs. In contrast, single-modality models tend to keep these manifolds "well-shaped" and concentrated.

Proposed Method: Bi-Directional Random Walk

To address this, the researchers propose a bi-directional random walk scheme to mine relationships on "heterogeneous manifolds." Rather than relying solely on a distorted joint feature space, this method treats the joint space as a "bridge" to connect cleaner, modality-specific manifolds.

Overview of the proposed bi-directional random walk scheme on heterogeneous manifolds.

Key Components:

  1. Graph Construction: A large graph is built where nodes represent images. Intra-modal edges ($A_{xx}, A_{yy}$) are calculated using modality-specific models to preserve reliable structures. Cross-modal edges ($A_{xy}$) are calculated using a joint "bridge" model.
  2. Random Walk (Diffusion): Starting from a query, a random walk algorithm spreads probability across the graph. This allows the search to find true positives that might be distant in the joint space but are close neighbors on the modality-specific manifold.
  3. Gallery-Guided Truncation: To maintain efficiency, the search uses a few cross-modal neighbors as "anchors" to explore the target manifold locally rather than searching the entire gallery.
  4. Bi-directional Retrieval: Similarity scores are calculated in both directions (Query $\to$ Gallery and Gallery $\to$ Query) and ensembled to produce a final ranking, which helps smooth out noise from cross-modal mapping.

Experimental Results

The proposed method was evaluated on digit retrieval (MNIST and SVHN) and person re-identification (RegDB and SYSU-MM01). In all cases, the manifold mining approach significantly outperformed baseline uniform-space models:

  • Digit Retrieval: On the MNIST $\to$ SVHN task, Mean Average Precision (mAP) increased from 46.7% to 60.3%.
  • Person Re-ID: On the SYSU-MM01 dataset, mAP improved from 26.7% to 38.0%.

Conclusion

This research demonstrates that the joint feature space in cross-modal retrieval is best utilized as a bridge between high-quality, modality-specific manifolds rather than a final destination. While the method relies on the initial quality of the cross-modal bridge, it offers a robust way to recover the natural structure of unseen classes in zero-shot settings. Future work may involve combining this manifold mining with generative data augmentation to further bridge the modality gap.

Summary updated

Abstract

Most recent approaches for the zero-shot cross-modal image retrieval map images from different modalities into a uniform feature space to exploit their relevance by using a pre-trained model. Based on the observation that manifolds of zero-shot images are usually deformed and incomplete, we argue that the manifolds of unseen classes are inevitably distorted during the training of a two-stream model that simply maps images from different modalities into a uniform space. This issue directly leads to poor cross-modal retrieval performance. We propose a bi-directional random walk scheme to mining more reliable relationships between images by traversing heterogeneous manifolds in the feature space of each modality. Our proposed method benefits from intra-modal distributions to alleviate the interference caused by noisy similarities in the cross-modal feature space. As a result, we achieved great improvement in the performance of the thermal v.s. visible image retrieval task. The code of this paper: https://github.com/fyang93/cross-modal-retrieval
BibTeX
@inproceedings{yang2020mining,
  author = {Fan Yang and Zheng Wang and Jing Xiao and Shin'ichi Satoh},
  title = {{Mining on Heterogeneous Manifolds for Zero-Shot Cross-Modal Image Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6949},
}
Figure