Introduction
Standard cross-modal retrieval models often struggle with "zero-shot" scenarios, where the categories of objects in the test set were never seen during training. The primary challenge lies in mapping different modalities (e.g., visible RGB and thermal infrared) into a single, uniform feature space. While this approach works for training data, it frequently distorts the natural structure—or manifolds—of unseen classes, leading to fragmented feature distributions and poor retrieval performance.
The Problem: Manifold Distortion
In cross-modal retrieval, the goal is to find a match for a query in one modality within a gallery of another. Current "two-stream" models attempt to reconcile conflicting relationships between modalities during training. For example, a lemon might be closer to a mango in color but closer to a lime in shape. Forcing these relationships into one space can cause the feature manifold of an unseen class to become cluttered and disconnected.
As shown in the t-SNE visualizations above, joint models can "shatter" clusters of unseen data into isolated blobs. In contrast, single-modality models tend to keep these manifolds "well-shaped" and concentrated.
Proposed Method: Bi-Directional Random Walk
To address this, the researchers propose a bi-directional random walk scheme to mine relationships on "heterogeneous manifolds." Rather than relying solely on a distorted joint feature space, this method treats the joint space as a "bridge" to connect cleaner, modality-specific manifolds.
Key Components:
- Graph Construction: A large graph is built where nodes represent images. Intra-modal edges ($A_{xx}, A_{yy}$) are calculated using modality-specific models to preserve reliable structures. Cross-modal edges ($A_{xy}$) are calculated using a joint "bridge" model.
- Random Walk (Diffusion): Starting from a query, a random walk algorithm spreads probability across the graph. This allows the search to find true positives that might be distant in the joint space but are close neighbors on the modality-specific manifold.
- Gallery-Guided Truncation: To maintain efficiency, the search uses a few cross-modal neighbors as "anchors" to explore the target manifold locally rather than searching the entire gallery.
- Bi-directional Retrieval: Similarity scores are calculated in both directions (Query $\to$ Gallery and Gallery $\to$ Query) and ensembled to produce a final ranking, which helps smooth out noise from cross-modal mapping.
Experimental Results
The proposed method was evaluated on digit retrieval (MNIST and SVHN) and person re-identification (RegDB and SYSU-MM01). In all cases, the manifold mining approach significantly outperformed baseline uniform-space models:
- Digit Retrieval: On the MNIST $\to$ SVHN task, Mean Average Precision (mAP) increased from 46.7% to 60.3%.
- Person Re-ID: On the SYSU-MM01 dataset, mAP improved from 26.7% to 38.0%.
Conclusion
This research demonstrates that the joint feature space in cross-modal retrieval is best utilized as a bridge between high-quality, modality-specific manifolds rather than a final destination. While the method relies on the initial quality of the cross-modal bridge, it offers a robust way to recover the natural structure of unseen classes in zero-shot settings. Future work may involve combining this manifold mining with generative data augmentation to further bridge the modality gap.
Summary updated