Fusion-Supervised Deep Cross-Modal Hashing

Li Wang, Lei Zhu, En Yu, Jiande Sun, Huaxiang Zhang

Abstract

Cross-modal retrieval aims to bridge the gap between different data modalities, such as images and text, to enable efficient searching across types. We present Fusion-supervised Deep Cross-modal Hashing (FDCH), a novel framework that addresses the heterogeneity and semantic gaps inherent in this task. FDCH employs a two-stage approach: first learning unified hash codes from fused image-text pairs to capture rich semantic correlations, and then using these codes to supervise the training of modality-specific networks. Experimental results on benchmark datasets demonstrate that FDCH significantly outperforms existing state-of-the-art methods in retrieval accuracy and efficiency.

Introduction

In the modern multi-modal digital landscape, the ability to perform cross-modal retrieval—such as finding relevant text descriptions for an image (I2T) or finding images based on a text query (T2I)—is increasingly critical. Cross-modal hashing has emerged as a leading solution, compressing high-dimensional data into compact binary strings to facilitate fast, storage-efficient searches.

However, traditional hashing methods face two primary challenges: the heterogeneity gap, where fundamentally different data structures (pixels vs. words) make it difficult to capture non-linear correlations, and the semantic gap, where binary codes may fail to represent the underlying meaning of the data. FDCH addresses these by leveraging a fusion-based supervision strategy.

The FDCH Framework

The FDCH architecture is defined by a two-stage "Fusion-supervised" learning process designed to maximize semantic consistency across modalities.

1. Unified Hash Code Learning

Rather than training separate networks from the outset, FDCH utilizes a Fusion Network. This network accepts features from both modalities—extracted via a CNN-F for images and a Bag-of-Words model for text—and processes them through a joint network to output a single, "unified" binary code for each pair. By observing both modalities simultaneously, the model learns a representation that captures shared semantic meaning more effectively than isolated training.

2. Modality-specific Training

The unified codes generated in the first stage serve as the "ground truth" labels for training individual image and text networks. During inference, only the network corresponding to the query modality is required, ensuring high efficiency.

The FDCH architecture: Stage 1 learns unified hash codes via a fusion network, which then supervise the training of modality-specific networks in Stage 2.

3. One-Stream Framework

The training process integrates three critical components: pair-wise similarity to ensure similar items have similar codes, classification information to maintain semantic consistency, and direct discrete optimization. Unlike methods that "relax" binary constraints into continuous values—which often introduces quantization errors—FDCH optimizes discrete binary codes directly.

Experimental Evaluation

FDCH was evaluated on two widely recognized benchmarks:

  • MIRFLICKR-25K: Comprising 20,015 image-text pairs with at least 20 labels.
  • NUS-WIDE: A large-scale dataset where a subset of the 10 most frequent concepts (186,577 pairs) was utilized.

Key Results

FDCH demonstrated superior performance across both datasets. On MIRFLICKR-25K, FDCH achieved a mean Average Precision (mAP) of 0.7762 for 16-bit codes in Image-to-Text retrieval, a substantial improvement over the previous state-of-the-art, DCMH (0.7050).

Precision-Recall curves for FDCH and baseline methods on the MIRFLICKR-25K dataset across different bit lengths and retrieval directions.

As shown in the Precision-Recall curves, FDCH maintains higher precision at higher recall levels compared to baselines such as LSSH, CMFH, and SePH. These gains are consistent across various code lengths (16, 32, and 64 bits) and both retrieval directions (I2T and T2I).

Conclusion

FDCH introduces a powerful paradigm for cross-modal hashing by using fusion as a teacher. By distilling knowledge from a joint fusion network into individual modality networks and employing direct discrete optimization, the framework effectively bridges the semantic and heterogeneity gaps. Future work will focus on reducing the dependency on perfectly paired training data and exploring scalability to additional modalities such as video and audio.

Summary updated

BibTeX
@inproceedings{wang2019fusion,
  author = {Li Wang and Lei Zhu and En Yu and Jiande Sun and Huaxiang Zhang},
  title = {{Fusion-Supervised Deep Cross-Modal Hashing}},
  booktitle = {ICME},
  year = {2019},
  url = {https://openreview.net/forum?id=pusXOQ5Ly7r},
}
Figure