Abstract
With the increasing demand of massive multimodal data storage and organization, cross-modal retrieval based on hashing technique has drawn much attention nowadays. It takes the binary codes of one modality as the query to retrieve the relevant hashing codes of another modality. However, the existing binary constraint makes it difficult to find the optimal cross-modal hashing function. Most approaches choose to relax the constraint and perform thresholding strategy on the real-value representation instead of directly solving the original objective. In this paper, we first provide a concrete analysis about the effectiveness of multimodal networks in preserving the inter- and intra-modal consistency. Based on the analysis, we provide a so-called Deep Binary Reconstruction (DBRC) network that can directly learn the binary hashing codes in an unsupervised fashion. The superiority comes from a proposed simple but efficient activation function, named as Adaptive Tanh (ATanh). The ATanh function can adaptively learn the binary codes and be trained via back-propagation. Extensive experiments on three benchmark datasets demonstrate that DBRC outperforms several state-of-the-art methods in both image2text and text2image retrieval task.
Introduction
Cross-modal retrieval aims to bridge the gap between different data modalities, such as images and text, allowing users to query one using the other. Hashing has emerged as a dominant technique due to its efficiency in both storage and search speed. By mapping high-dimensional data into a compact binary space, retrieval becomes a matter of calculating Hamming distances, which is computationally inexpensive.
However, a significant hurdle remains: the discrete nature of binary codes. Standard neural networks rely on backpropagation, which requires differentiable functions. The "sign" function used for binarization is non-differentiable, leading many researchers to adopt a "relax-then-threshold" strategy. In this approach, the network is trained to output real values, which are then quantized into binary codes post-hoc. This relaxation often results in a significant "binary gap," where the semantic relationships learned in the continuous space are lost during quantization.
The DBRC Framework
To address these limitations, we propose the Deep Binary Reconstruction (DBRC) network. DBRC is an unsupervised framework designed to learn binary hashing codes directly. The architecture consists of two main components: a multimodal encoder-decoder structure and a specialized hashing layer.
The core idea is to learn a joint representation in a real-valued space and then transform it into a binary space using a novel activation function. This binary representation is then used to reconstruct the original features of both modalities. By minimizing the reconstruction error, the network ensures that the binary codes capture the most salient features necessary to represent both the image and text data.
Adaptive Tanh (ATanh) Activation
The primary innovation of DBRC is the Adaptive Tanh (ATanh) activation function. Traditional Tanh functions provide a smooth approximation of the sign function but do not enforce binary outputs strictly. ATanh introduces a learnable scaling parameter, $\alpha$, for each bit:
$$f(j) = \tanh(\alpha_j \cdot j)$$
During the early stages of training, $\alpha$ is kept small, allowing the function to behave like a standard, smooth Tanh. As training progresses, $\alpha$ increases, sharpening the function until it approximates the step-like behavior of the sign function. Because ATanh remains differentiable throughout this process, the network can optimize the binary codes directly via backpropagation, effectively bridging the binary gap.
Experimental Results
We evaluated DBRC on three benchmark datasets: Wiki, FLICKR-25K, and NUS-WIDE. We compared its performance against several state-of-the-art methods, including linear models like CMFH and LSSH, and deep learning models like Corr-Full-AE.
As shown in the results table, DBRC consistently outperforms competing methods across both Image-to-Text (I2T) and Text-to-Image (T2I) tasks. Notably, on the Wiki dataset, DBRC achieves a MAP of 0.5439 for 16-bit T2I retrieval, a substantial improvement over the 0.4992 achieved by LSSH. Furthermore, DBRC demonstrates remarkable stability as the code length increases. While some methods see performance degradation due to the difficulty of optimizing longer binary strings, DBRC maintains or improves its precision, highlighting the effectiveness of the ATanh-driven optimization.
Conclusion
Deep Binary Reconstruction offers a robust solution to the quantization challenges in cross-modal hashing. By integrating the ATanh activation function into an unsupervised reconstruction framework, DBRC avoids the pitfalls of traditional relaxation methods. The result is a more semantically rich binary representation that significantly improves retrieval accuracy across diverse multimodal datasets.
Summary updated