Deep Binary Reconstruction for Cross-modal Hashing

Xuelong Li, Di Hu, Feiping Nie

TL;DR

Deep Binary Reconstruction (DBRC) is a novel unsupervised framework that directly learns binary hashing codes for cross-modal retrieval (like searching for images using text). By introducing a differentiable Adaptive Tanh (ATanh) activation function, the model avoids the common "relax-then-threshold" pitfall, allowing the network to optimize for binary outputs directly through backpropagation.

What Problem Does This Paper Address?

In cross-modal retrieval, the goal is to find relevant items in one modality (e.g., an image) using a query from another (e.g., a text description). Hashing is the preferred method for this because it converts high-dimensional data into short binary strings, making storage and search incredibly fast.

However, training a neural network to output binary values (0s and 1s, or -1s and 1s) is mathematically difficult because the "sign" function is non-differentiable—you can't run backpropagation through it. Most existing methods solve this by "relaxing" the constraint: they train the network to output real numbers and then simply round them to the nearest binary value after training is done. This "two-stage" approach is sub-optimal because the network never actually learns how to be "binary-efficient" during training, leading to a loss of semantic information.

Methods

The authors propose Deep Binary Reconstruction (DBRC), which uses an encoder-decoder architecture to learn a shared binary space between modalities.

  1. Multimodal Foundation: The authors first provide a theoretical analysis of Multimodal Restricted Boltzmann Machines (MRBMs), proving that a shared hidden layer can effectively preserve both intra-modal (within one type of data) and inter-modal (across different types) consistency.
  2. Adaptive Tanh (ATanh): This is the paper's "secret sauce." Instead of a standard Tanh or a hard Sign function, they use: $$f(s) = \tanh(\alpha s)$$ where $\alpha$ is a learnable scaling parameter.

  3. When $\alpha$ is small, it behaves like a smooth, differentiable Tanh.

  4. As $\alpha$ increases, the function "sharpens" until it closely approximates the Sign function.
  5. Direct Optimization: Because ATanh is differentiable, the network can be trained end-to-end. They include a regularization term to control the growth of $\alpha$, ensuring the network gradually "learns" to become binary.
  6. Unsupervised Reconstruction: The model is trained to take one modality (or both), compress it into a shared binary hashing layer, and then reconstruct the original high-dimensional data for both modalities. This forces the binary codes to capture the most essential features of the data.

Datasets

The model was evaluated on three standard benchmark datasets for cross-modal retrieval:

  • Wiki: 2,866 image-text pairs from Wikipedia "featured articles," categorized into 10 topics.
  • FLICKR-25K: 25,000 images from Flickr associated with various textual tags.
  • NUS-WIDE: A large-scale dataset of nearly 270,000 images; the authors used a subset of 4,000 image-tag pairs for evaluation across 10 common concepts.

Key Insights and Ideas

  • The "Binary Gap" is Real: The paper highlights that traditional two-stage methods destroy the learned semantic space during the final binarization step. DBRC proves that training with the binary constraint in mind yields much more robust codes.
  • Learnable Quantization: Instead of forcing a global threshold, ATanh allows the network to learn a specific $\alpha$ for each bit in the hash code. This makes the hashing layer more adaptable to the specific structure of the data.
  • Reconstruction as a Proxy for Similarity: By forcing the network to reconstruct the original data from a shared binary bottleneck, the model naturally discovers the correlations between text and images without needing explicit labels (unsupervised).

Key Results

DBRC was compared against several state-of-the-art linear (e.g., CMFH, LSSH) and nonlinear (e.g., Corr-Full-AE, DMHOR) methods.

  • Superior Accuracy: DBRC consistently achieved the highest Mean Average Precision (MAP) across almost all datasets and code lengths (16, 32, 64, and 128 bits).
  • Text-to-Image (T2I) Performance: On the Wiki dataset, DBRC achieved a MAP of 0.5439 for 16-bit codes, significantly outperforming the next best method (LSSH at 0.4992).
  • Stability: Unlike other deep learning methods that saw performance drop as code length increased (due to sparse Hamming space), DBRC remained stable, showing its effectiveness in learning high-dimensional binary representations.

Limitations and Future Work

  • Final Binarization: While ATanh gets very close to binary values, the authors admit that a tiny fraction of activations still fall between -1 and 1 at the end of training, requiring a final (though much less harmful) rounding step.
  • Computational Overhead: As an encoder-decoder model, training might be more computationally intensive than simpler linear methods, though the authors note the complexity of the ATanh function itself is negligible.
  • Scalability: While tested on NUS-WIDE, the experiments used a sampled subset. Testing on the full multi-million-scale datasets would be a logical next step to prove real-world scalability.

Summary updated

Abstract

With the increasing demand of massive multimodal data storage and organization, cross-modal retrieval based on hashing technique has drawn much attention nowadays. It takes the binary codes of one modality as the query to retrieve the relevant hashing codes of another modality. However, the existing binary constraint makes it difficult to find the optimal cross-modal hashing function. Most approaches choose to relax the constraint and perform thresholding strategy on the real-value representation instead of directly solving the original objective. In this paper, we first provide a concrete analysis about the effectiveness of multimodal networks in preserving the inter- and intra-modal consistency. Based on the analysis, we provide a so-called Deep Binary Reconstruction (DBRC) network that can directly learn the binary hashing codes in an unsupervised fashion. The superiority comes from a proposed simple but efficient activation function, named as Adaptive Tanh (ATanh). The ATanh function can adaptively learn the binary codes and be trained via back-propagation. Extensive experiments on three benchmark datasets demonstrate that DBRC outperforms several state-of-the-art methods in both image2text and text2image retrieval task.
BibTeX
@inproceedings{li2017deep,
  author = {Xuelong Li and Di Hu and Feiping Nie},
  title = {{Deep Binary Reconstruction for Cross-modal Hashing}},
  booktitle = {ACM Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=RcbERc8kS39},
}

Loading related content...

Figure