Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval

Rukai Wei, Heng Cui, Yu Liu, Yanzhao Xie, Yufeng Hou, Ke Zhou

Abstract

Implementing cross-modal hashing between 2D images and 3D point-cloud data is a growing concern in real-world retrieval systems. Simply applying existing cross-modal approaches to this new task fails to adequately capture latent multi-modal semantics and effectively bridge the modality gap between 2D and 3D. To address these issues without relying on hand-crafted labels, we propose contrastive masked autoencoders based self-supervised hashing (CMAH) for retrieval between images and point-cloud data. We start by contrasting 2D-3D pairs and explicitly constraining them into a joint Hamming space. This contrastive learning process ensures robust discriminability for the generated hash codes and effectively reduces the modality gap. Moreover, we utilize multi-modal auto-encoders to enhance the model's understanding of multi-modal semantics. By completing the masked image/point-cloud data modeling task, the model is encouraged to capture more localized clues. In addition, the proposed multi-modal fusion block facilitates fine-grained interactions among different modalities. Extensive experiments on three public datasets demonstrate that the proposed CMAH significantly outperforms all baseline methods.

Introduction

In fields such as autonomous driving and robotics, the ability to rapidly retrieve 3D point cloud models from 2D images (and vice versa) is increasingly vital. Cross-modal hashing has emerged as a preferred solution, as it maps high-dimensional data into compact binary codes, enabling efficient similarity searches via Hamming distance. However, the transition from 2D pixels to 3D coordinates presents a significant "modality gap." Unlike text-image retrieval, 3D point clouds are irregular and unordered, making it difficult for standard self-supervised methods to capture fine-grained geometric semantics. Existing approaches often fail to align these disparate modalities effectively, leading to suboptimal retrieval performance.

Methodology

The proposed Contrastive Masked Auto-encoders based Hashing (CMAH) framework addresses these challenges through a dual-pronged approach: global relationship preservation and local feature capture.

The CMAH architecture featuring dual-stream encoders, a multi-modal fusion block, and masked auto-encoding reconstruction.

Dual-Stream Encoding and Contrastive Learning

CMAH utilizes two transformer-based encoders: a Vision Transformer (ViT) for 2D images and a Point-MAE style encoder for 3D point clouds. To bridge the modality gap, the model employs multi-modal contrastive learning. It aligns "positive" pairs (corresponding 2D and 3D representations of the same object) in a shared Hamming space while pushing "negative" pairs apart. A unique aspect of CMAH is the use of "masked-full" contrastive pairs, which forces the model to learn modality-invariant features that are robust to local noise and occlusions.

Masked Auto-encoding and Multi-modal Fusion

To enhance the understanding of local semantics, CMAH incorporates a Masked Auto-encoder (MAE) task. During training, 75% of image patches and 60% of point cloud tokens are randomly masked. The model is then tasked with reconstructing these missing components.

The Multi-modal Fusion Block is central to the architecture. Positioned between the encoders and decoders, this block uses a co-attention mechanism that allows one modality to serve as a "prompt" for the other. For instance, features from a 2D image can guide the reconstruction of masked 3D point cloud tokens, ensuring that the encoders learn a truly integrated multi-modal representation.

Experimental Results

The performance of CMAH was evaluated on three benchmark datasets: ShapeNetRender, ModelNet, and ShapeNet-55. These datasets provide a diverse range of 3D objects and rendered 2D viewpoints for comprehensive testing.

Performance Comparison

CMAH significantly outperforms state-of-the-art (SOTA) self-supervised hashing methods, such as DJSRH and ASSPH. On the ModelNet dataset for the Image-to-Point Cloud (I → P) task, CMAH achieved a Mean Average Precision (mAP) improvement of 12.3% over the previous best method.

Precision vs. Top@K curves on ShapeNetRender and ModelNet datasets, demonstrating CMAH's superior performance over baseline methods.

Key Insights

  • Masking as Augmentation: Masking serves as a powerful data augmentation strategy. By matching partial data in one modality to full data in another, the model focuses on core semantic features rather than redundant background information.
  • Efficiency: Despite its sophisticated training regime, CMAH remains highly efficient for real-time applications. Encoding times are approximately 10.3ms for images and 13.4ms for point clouds on standard hardware.
  • Robustness at Low Bit Lengths: CMAH maintains high precision even with short hash codes (e.g., 16-bit), which is essential for systems with limited storage or memory constraints.

Conclusion

CMAH represents a significant step forward in self-supervised 2D-3D cross-modal retrieval. By combining global contrastive learning with local masked reconstruction and fine-grained multi-modal fusion, it effectively bridges the modality gap without requiring manual labels. Future research will focus on reducing the dependency on pre-trained backbones and extending the framework to handle more complex, "in-the-wild" datasets with varying lighting and occlusion conditions.

Summary updated

Abstract

Implementing cross-modal hashing between 2D images and 3D point-cloud data is a growing concern in real-world retrieval systems. Simply applying existing cross-modal approaches to this new task fails to adequately capture latent multi-modal semantics and effectively bridge the modality gap between 2D and 3D. To address these issues without relying on hand-crafted labels, we propose contrastive masked autoencoders based self-supervised hashing (CMAH) for retrieval between images and point-cloud data. We start by contrasting 2D-3D pairs and explicitly constraining them into a joint Hamming space. This contrastive learning process ensures robust discriminability for the generated hash codes and effectively reduces the modality gap. Moreover, we utilize multi-modal auto-encoders to enhance the model’s understanding of multi-modal semantics. By completing the masked image/point-cloud data modeling task, the model is encouraged to capture more localized clues. In addition, the proposed multi-modal fusion block facilitates fine-grained interactions among different modalities. Extensive experiments on three public datasets demonstrate that the proposed CMAH significantly outperforms all baseline methods.
BibTeX
@misc{wei2024contrastive,
  author = {Rukai Wei and Heng Cui and Yu Liu and Yanzhao Xie and Yufeng Hou and Ke Zhou},
  title = {{Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval}},
  year = {2024},
  eprint = {2408.05711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05711},
}
Figure