Semi-supervised Multimodal Hashing

Dayong Tian, Maoguo Gong, Deyun Zhou, Jiao Shi, Yu LeiarXiv 2017 · OpenReview

TL;DR

This paper introduces a novel semi-supervised multimodal hashing (SSMH) method that effectively leverages a small amount of labeled data to improve retrieval accuracy across different data modalities (like images and text). By using fuzzy logic to estimate labels for unlabeled data, SSMH bridges the gap between unsupervised methods (which are less accurate) and fully supervised methods (which require excessive labeled data).

What Problem Does This Paper Address?

The digital world is awash with correlated data across different modalities – think images and their accompanying text captions on social media, or videos and their tags. Efficiently retrieving relevant information across these modalities is a major challenge due to the sheer volume of data. Multimodal hashing offers a solution by compressing data into compact binary codes, enabling faster searches and reduced storage. However, existing methods face a dilemma: unsupervised hashing methods often lack accuracy, while supervised methods demand a prohibitive amount of manually labeled data, making them impractical for large-scale applications. This paper tackles this gap by proposing a semi-supervised approach that strikes a balance, requiring significantly less labeled data than fully supervised methods while achieving competitive performance.

Methods

The core of the proposed SSMH method lies in its two-stage approach:

  1. Supervised Hashing for Label Estimation: The method first treats the available labeled data as a special modality. It then employs a supervised hashing technique to learn transformation matrices. These matrices are used to generate initial "candidate" labels for the unlabeled data. This stage is crucial for inferring semantic relationships even when explicit labels are missing.
  2. Fuzzy Logic-Based Label Estimation and Hashing: Here's where the novelty truly shines. Fuzzy logic is introduced to estimate a more robust label matrix for the unlabeled data. By incorporating a "membership variable," the method quantifies the probability that a transformed unlabeled data point's label aligns with a particular modality's learned representation. This fuzzy approach, inspired by fuzzy c-means clustering, allows for a more nuanced estimation of labels compared to hard assignments. Once these estimated labels are refined, a modified supervised hashing function is used to generate the final unified binary codes for all modalities.

Datasets

The experiments were conducted on the MIRFlickr dataset. This dataset contains image-text pairs, with images represented by GIST descriptors and text by feature vectors derived from PCA on textual tags. The authors created three partially labeled data sets by using 10%, 50%, and 90% of the available labels for training.

Key Insights and Ideas

  • Fuzzy Logic as a Bridge: The paper demonstrates that fuzzy logic can effectively bridge the gap between limited labeled data and the need for accurate semantic representation in multimodal hashing. It allows for a more flexible and robust estimation of labels for unlabeled data.
  • Semi-Supervised Advantage: The results highlight the significant advantage of semi-supervised learning in this domain. By intelligently using a small fraction of labels, SSMH achieves performance comparable to, or even exceeding, fully supervised methods that use a much larger proportion of labeled data.
  • Balancing Accuracy and Data Requirements: SSMH offers a practical solution for real-world scenarios where obtaining extensive labeled data is a bottleneck. It provides a good trade-off between retrieval accuracy and the cost of data annotation.

Key Results

The proposed SSMH method demonstrated promising results on the MIRFlickr dataset:

  • With 10% labels, SSMH achieved performance comparable to the worst-performing fully supervised methods.
  • With 50% labels, SSMH achieved a medium performance among the compared supervised methods.
  • With 90% labels, SSMH's performance was comparable to the best supervised method (MDBE), with differences not being statistically significant.

This indicates that SSMH scales well with increasing amounts of labeled data, consistently outperforming or matching supervised methods that require significantly more annotations.

Limitations and Future Work

  • Parameter Sensitivity: While the paper mentions setting parameters like $\alpha$ and $\gamma$ to specific values, the sensitivity of the method to these parameters and the optimal strategy for tuning them for different datasets are not extensively discussed.
  • Computational Complexity: Although the paper aims to address the scalability issues of previous methods, the iterative nature of the fuzzy label estimation and hashing process might still pose computational challenges for extremely large datasets.
  • Generalizability: While tested on MIRFlickr, further evaluation on diverse multimodal datasets would be beneficial to assess the generalizability of SSMH.

The authors suggest that future work could explore more advanced fuzzy logic techniques or investigate different strategies for incorporating label information to further enhance the performance of semi-supervised multimodal hashing.

Summary updated

Abstract

Retrieving nearest neighbors across correlated data in multiple modalities, such as image-text pairs on Facebook and video-tag pairs on YouTube, has become a challenging task due to the huge amount of data. Multimodal hashing methods that embed data into binary codes can boost the retrieving speed and reduce storage requirement. As unsupervised multimodal hashing methods are usually inferior to supervised ones, while the supervised ones requires too much manually labeled data, the proposed method in this paper utilizes a part of labels to design a semi-supervised multimodal hashing method. It first computes the transformation matrices for data matrices and label matrix. Then, with these transformation matrices, fuzzy logic is introduced to estimate a label matrix for unlabeled data. Finally, it uses the estimated label matrix to learn hashing functions for data in each modality to generate a unified binary code matrix. Experiments show that the proposed semi-supervised method with 50% labels can get a medium performance among the compared supervised ones and achieve an approximate performance to the best supervised method with 90% labels. With only 10% labels, the proposed method can still compete with the worst compared supervised one.
BibTeX
@misc{tian2017semi,
  author = {Dayong Tian and Maoguo Gong and Deyun Zhou and Jiao Shi and Yu Lei},
  title = {{Semi-supervised Multimodal Hashing}},
  year = {2017},
  eprint = {1712.03404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.03404},
}
Figure