Discriminative Cross-View Binary Representation Learning

Liu Liu, Hairong Qi

TL;DR

This paper introduces Discriminative Cross-View Hashing (DCVH), an end-to-end deep learning framework that learns compact binary representations for multimedia data (images and text). DCVH excels at aligning these different modalities by minimizing Hamming distance, leading to improved performance in cross-view retrieval, single-view retrieval, and image annotation tasks.

What Problem Does This Paper Address?

The explosion of multimedia content (images, text, videos) necessitates efficient ways to search and understand this data. A key challenge is learning compact, discriminative representations that capture the semantic meaning of different modalities and allow for effective cross-modal retrieval (e.g., searching for images using text queries). Existing cross-view hashing methods often rely on explicit similarity matrices, which are computationally expensive and may not preserve discriminative information crucial for other tasks like image annotation. This paper aims to bridge this gap by proposing a unified, end-to-end approach that learns discriminative binary representations for multiple views simultaneously.

Methods

DCVH employs a novel end-to-end architecture with several key components:

  • CNN-based Nonlinear Projections: For both image and text views, DCVH utilizes Convolutional Neural Networks (CNNs) to project the data into a latent feature space.
  • Direct Binary Embedding (DBE) Layers: These layers are integrated into the CNNs, allowing for the learning of continuous binary-like representations that approximate the final binary codes without explicit quantization loss during training. This simplifies the training process and avoids common issues with discrete optimization.
  • Multilabel Classification for Discriminative Learning: Instead of relying solely on cross-view similarities, DCVH leverages multilabel classification as a supervision signal. This encourages the learned binary codes to be discriminative, which is beneficial for various tasks.
  • Hamming Distance Minimization for View Alignment: To align the binary representations from different views (images and texts), DCVH directly minimizes the Hamming distance between them. This is efficiently achieved using a bit-wise XOR operation, which is well-suited for binary codes.

Datasets

The proposed DCVH was evaluated on two widely-used image-text benchmark datasets:

  • MS COCO: A large-scale dataset for image recognition, segmentation, and captioning, containing 83K training and 40K validation images with multiple annotations per image.
  • MIRFLICKR: A dataset of 25K color images, each associated with several textual tags.

The authors also mention using pre-trained GloVe vectors for text representation and ResNet-50 for image feature extraction.

Key Insights and Ideas

  • Discriminative Power of Classification: The paper highlights that learning binary representations through classification tasks, rather than just similarity, leads to more discriminative codes that are useful for a wider range of tasks.
  • End-to-End Learning for Multimedia: DCVH demonstrates the effectiveness of an end-to-end approach for cross-modal learning, avoiding the need for hand-crafted features or separate training stages for different components.
  • Efficient View Alignment: The use of bit-wise XOR for Hamming distance minimization offers an efficient and effective way to align representations from different modalities, directly optimizing for semantic correspondence.
  • Multitasking Binary Representations: The learned binary representations from DCVH are shown to be versatile, performing well not only in cross-view retrieval but also in single-view retrieval and image annotation tasks, underscoring their rich semantic content.

Key Results

  • Superior Cross-View Retrieval: DCVH consistently outperforms state-of-the-art cross-view hashing methods (including its own variations DCVH-THN and DCVH-CVH) on both MS COCO and MIRFLICKR datasets, achieving higher mean Average Precision (mAP) across various code lengths. For instance, on MS COCO, DCVH improves mAP by up to 12% compared to its variations in text query retrieval.
  • Competitive Single-View Retrieval: DCVH also shows strong performance in single-view image retrieval, outperforming methods like HashNet by around 2.5% and DBE by about 5% on the MS COCO dataset.
  • Effective Image Annotation: For image annotation on MS COCO, DCVH achieves competitive results, demonstrating its ability to learn multitask binary representations.
  • Hyperparameter Sensitivity: The study on the hyperparameter $\lambda$ (controlling view alignment strength) reveals a trade-off: too much alignment can hurt discriminativeness, while too little can impair cross-view performance. An optimal value of $\lambda=0.2$ was found to balance these aspects for various tasks.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper explores the impact of $\lambda$, finding the optimal value for different datasets and tasks might require further investigation.
  • Extensibility to More Views: The paper briefly discusses extending DCVH to handle more than two views, suggesting a general framework for mutual Hamming distance minimization. This could be a promising direction for future research.
  • Computational Cost: Although end-to-end, deep learning models can be computationally intensive. Further optimizations for training and inference speed might be beneficial.

Summary updated

Abstract

Learning compact representation is vital and challenging for large scale multimedia data. Cross-view/crossmodal hashing for effective binary representation learning has received significant attention with exponentially growing availability of multimedia content. Most existing crossview hashing algorithms emphasize the similarities in individual views, which are then connected via cross-view similarities. In this work, we focus on the exploitation of the discriminative information from different views, and propose an end-to-end method to learn semantic-preserving and discriminative binary representation, dubbed Discriminative Cross-View Hashing (DCVH), in light of learning multitasking binary representation for various tasks including cross-view retrieval, image-to-image retrieval, and image annotation/tagging. The proposed DCVH has the following key components. First, it uses convolutional neural network (CNN) based nonlinear hashing functions and multilabel classification for both images and texts simultaneously. Such hashing functions achieve effective continuous relaxation during training without explicit quantization loss by using Direct Binary Embedding (DBE) layers. Second, we propose an effective view alignment via Hamming distance minimization, which is efficiently accomplished by bit-wise XOR operation. Extensive experiments on two image-text benchmark datasets demonstrate that DCVH outperforms state-of-the-art cross-view hashing algorithms as well as single-view image hashing algorithms. In addition, DCVH can provide competitive performance for image annotation/tagging.
BibTeX
@inproceedings{liu2018discriminative,
  author = {Liu Liu and Hairong Qi},
  title = {{Discriminative Cross-View Binary Representation Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2018.00193},
}
Figure