Semantic Topic Multimodal Hashing for Cross-Media Retrieval

Di Wang, Xinbo Gao, Xiumei Wang, Lihuo HeIJCAI 2015

Abstract

Cross-media retrieval, such as searching for images via text queries, is a critical task in large-scale data management. Traditional multimodal hashing methods often suffer from quantization errors due to continuous relaxation and fail to capture high-level semantic correlations. This paper presents Semantic Topic Multimodal Hashing (STMH), a framework that directly learns discrete binary codes by discovering latent semantic topics in text and robust concepts in images. By leveraging $L_{2,1}$-norm matrix factorization and avoiding relaxation, STMH achieves superior retrieval accuracy and scalability on benchmark datasets like Wiki and NUS-WIDE.

Introduction

The explosion of multimodal data on the web has necessitated efficient cross-media retrieval techniques. Multimodal hashing has emerged as a dominant solution, mapping heterogeneous data into a common low-dimensional Hamming space. However, existing methods often overlook the semantic gap between low-level features and high-level concepts, or introduce significant errors by relaxing discrete constraints during optimization. STMH addresses these issues by treating hashing as a discrete problem from the outset and focusing on latent semantic structures.

Challenges in Multimodal Hashing

The authors identify three primary limitations in current hashing techniques:

  1. The Relaxation Gap: Most algorithms solve a "relaxed" continuous problem and then binarize the result, leading to substantial quantization errors, especially as code length increases.
  2. Semantic Neglect: Many methods focus on statistical correlations of low-level features rather than the underlying semantic topics (e.g., linking a photo of a beach to the word "ocean").
  3. Computational Complexity: Graph-based approaches often scale poorly with the number of data points because they require building massive similarity matrices.

The STMH Framework

STMH addresses these challenges by modeling the latent structure of both text and images to create a unified semantic bridge.

The STMH framework discovers latent topics in text (e.g., natural vs. cultural landscapes) and latent concepts in images (e.g., mountains, architecture) to align different modalities.

Semantic Topic Modeling for Text

Instead of relying on raw bag-of-words vectors, STMH identifies latent topics within the text. Each bit in the resulting hash code corresponds to the presence of a specific semantic topic, ensuring that the code captures high-level meaning rather than just keyword frequency.

Robust Concept Modeling for Images

For image data, STMH employs Robust Matrix Factorization. A key innovation is the use of the $L_{2,1}$-norm loss function. Unlike standard squared error, the $L_{2,1}$-norm is significantly more robust to noise and outliers. This allows the model to focus on salient visual concepts while ignoring irrelevant background noise or artifacts common in web-scraped datasets.

Semantic Correlation Matching and Discrete Optimization

After extracting topics and concepts, STMH learns a correlation matrix to map these representations into a shared subspace. Unlike relaxation-based methods, STMH optimizes the binary codes directly through an iterative process. This eliminates quantization errors and ensures that the performance of the hash codes improves consistently as the code length increases, avoiding the performance plateaus seen in competing methods.

Experimental Evaluation

The framework was evaluated on two widely used benchmarks: the Wiki dataset (highly semantic image-text pairs) and the NUS-WIDE dataset (large-scale, noisy web data).

  • Accuracy: STMH consistently outperformed state-of-the-art baselines (e.g., CMSSH, CVH, LSSH) in Mean Average Precision (mAP) across all code lengths.
  • Scalability: The training complexity scales linearly with the number of samples, making it suitable for real-world applications involving millions of data points.
  • Long-Code Stability: While other methods often degrade with longer hash codes due to accumulated quantization error, STMH showed a consistent upward trend in precision as more bits were added.

Conclusion

Semantic Topic Multimodal Hashing (STMH) provides a robust and efficient solution for cross-media retrieval. By bridging the semantic gap through latent topic discovery and maintaining discrete constraints during optimization, it sets a new standard for multimodal hashing performance. Future work may explore deep learning integration to further enhance feature extraction before the hashing stage.

Summary updated

BibTeX
@inproceedings{wang2015semantic,
  author = {Di Wang and Xinbo Gao and Xiumei Wang and Lihuo He},
  title = {{Semantic Topic Multimodal Hashing for Cross-Media Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure