Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval

Fan Zhang, Xian-Sheng Hua, Chong Chen, Xiao Luo

Introduction

2D-3D cross-modal retrieval—the task of finding a 3D model that matches a 2D image—is a fundamental challenge in computer vision. The core difficulty lies in the "modality gap": 2D pixels and 3D point clouds are mathematically heterogeneous, making it difficult to align them in a shared embedding space. While supervised methods have made progress, they rely on massive amounts of expensive 2D-3D paired annotations. Semi-supervised learning offers a path forward by leveraging unlabeled data, but existing techniques often suffer from noisy pseudo-labels, leading to suboptimal performance.

To address these challenges, researchers have developed FIVE (Fine-grained Prototypical Voting with Heterogeneous Mixup), a novel semi-supervised framework designed to bridge the 2D-3D gap with high precision even when labeled data is scarce.

The FIVE Framework

The FIVE framework integrates three key components to improve cross-modal alignment and robustness: Fine-grained Prototypical Voting, Heterogeneous Mixup, and Bi-level Optimization. The overall architecture is designed to process 2D and 3D inputs through separate encoders, align them via a voting mechanism, and refine the shared space through feature blending and robust optimization.

The overall architecture of the FIVE framework, illustrating the flow from 2D/3D inputs to bi-level optimization.

Fine-grained Prototypical Voting

Traditional semi-supervised methods often represent a class using a single centroid. However, complex categories like "chairs" exhibit significant intra-class variation that a single point cannot capture. FIVE employs K-means clustering to generate multiple prototypes per class.

When processing an unlabeled sample, the model identifies the $M$ nearest prototypes across the embedding space. These prototypes act as a "panel of experts" that vote on the sample's label. This voting mechanism provides more robust pseudo-labels, particularly for samples near class boundaries, by accounting for the diverse geometric and visual manifestations of a category.

A comparison between traditional single centroid representation (a) and the proposed prototypical voting mechanism (b).

Heterogeneous Mixup and Bi-level Optimization

To further bridge the modality gap, FIVE introduces Heterogeneous Mixup. By blending 2D and 3D features that share the same pseudo-label, the model creates "virtual" cross-modal samples. This forces the encoders to learn modality-invariant representations, effectively smoothing the transition between 2D and 3D data in the latent space.

Finally, to mitigate the impact of inevitable pseudo-label noise, the framework utilizes Bi-level Optimization inspired by Sharpness-Aware Minimization (SAM). Instead of merely seeking the global minimum of the loss function, the model optimizes for "flatness" in the loss landscape. This ensures that small perturbations in the input or parameters—often caused by noisy labels—do not lead to significant performance degradation, enhancing the model's generalization capabilities.

Experimental Results

The FIVE framework was evaluated on three benchmark datasets: 3D MNIST, ModelNet10, and ModelNet40. The results demonstrate a significant leap over existing state-of-the-art (SOTA) methods, particularly in low-label regimes.

On the ModelNet40 dataset with only 800 labeled samples, FIVE achieved a 72.94% Mean Average Precision (mAP) for 2D-to-3D retrieval, outperforming the previous best method (RONO) by a substantial 22.10%. Similar improvements were observed on 3D MNIST, where FIVE surpassed the next best method by an average of 4.74%.

Qualitative analysis using t-SNE visualizations confirmed that FIVE produces tighter, more distinct clusters where 2D and 3D embeddings are well-intermingled, indicating successful cross-modal alignment.

Conclusion

FIVE introduces a robust approach to semi-supervised 2D-3D retrieval by moving beyond simple class centroids to a more nuanced voting mechanism. By combining this with heterogeneous feature mixup and sharpness-aware optimization, the framework effectively bridges the modality gap with minimal supervision. Future work will explore extending this approach to zero-shot retrieval and more complex, multi-object 3D scenes.

Summary updated

Abstract

This paper studies the problem of semi-supervised 2D-3D retrieval, which aims to align both labeled and unla-beled 2D and 3D data into the same embedding space. The problem is challenging due to the complicated heteroge-neous relationships between 2D and 3D data. Moreover, label scarcity in real-world applications hinders from gen-erating discriminative representations. In this paper, we propose a semi-supervised approach named Fine-grained Prototypcical ⊻oting with Heterogeneous Mixup (FIVE), which maps both 2D and 3D data into a common embed-ding space for cross-modal retrieval. Specifically, we gen-erate fine-grained prototypes to model intra-class variation for both 2D and 3D data. Then, considering each unlabeled sample as a query, we retrieve relevant prototypes to vote for reliable and robust pseudo-labels, which serve as guid-ance for discriminative learning under label scarcity. Fur-thermore, to bridge the semantic gap between two modali-ties, we mix cross-modal pairs with similar semantics in the embedding space and then perform similarity learning for cross-modal discrepancy reduction in a soft manner. The whole FIVE is optimized with the consideration of sharp-ness to mitigate the impact of potential label noise. Exten-sive experiments on benchmark datasets validate the supe-riority of FIVE compared with a range of baselines in differ-ent settings. On average, FIVE outperforms the second-best approach by 4.74% on 3D MNIST, 12.94% on ModelNet10, and 22.10% on ModelNet40.
BibTeX
@inproceedings{zhang2024fine,
  author = {Fan Zhang and Xian-Sheng Hua and Chong Chen and Xiao Luo},
  title = {{Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01610},
}
Figure