Introduction
2D-3D cross-modal retrieval—the task of finding a 3D model that matches a 2D image—is a fundamental challenge in computer vision. The core difficulty lies in the "modality gap": 2D pixels and 3D point clouds are mathematically heterogeneous, making it difficult to align them in a shared embedding space. While supervised methods have made progress, they rely on massive amounts of expensive 2D-3D paired annotations. Semi-supervised learning offers a path forward by leveraging unlabeled data, but existing techniques often suffer from noisy pseudo-labels, leading to suboptimal performance.
To address these challenges, researchers have developed FIVE (Fine-grained Prototypical Voting with Heterogeneous Mixup), a novel semi-supervised framework designed to bridge the 2D-3D gap with high precision even when labeled data is scarce.
The FIVE Framework
The FIVE framework integrates three key components to improve cross-modal alignment and robustness: Fine-grained Prototypical Voting, Heterogeneous Mixup, and Bi-level Optimization. The overall architecture is designed to process 2D and 3D inputs through separate encoders, align them via a voting mechanism, and refine the shared space through feature blending and robust optimization.
Fine-grained Prototypical Voting
Traditional semi-supervised methods often represent a class using a single centroid. However, complex categories like "chairs" exhibit significant intra-class variation that a single point cannot capture. FIVE employs K-means clustering to generate multiple prototypes per class.
When processing an unlabeled sample, the model identifies the $M$ nearest prototypes across the embedding space. These prototypes act as a "panel of experts" that vote on the sample's label. This voting mechanism provides more robust pseudo-labels, particularly for samples near class boundaries, by accounting for the diverse geometric and visual manifestations of a category.
Heterogeneous Mixup and Bi-level Optimization
To further bridge the modality gap, FIVE introduces Heterogeneous Mixup. By blending 2D and 3D features that share the same pseudo-label, the model creates "virtual" cross-modal samples. This forces the encoders to learn modality-invariant representations, effectively smoothing the transition between 2D and 3D data in the latent space.
Finally, to mitigate the impact of inevitable pseudo-label noise, the framework utilizes Bi-level Optimization inspired by Sharpness-Aware Minimization (SAM). Instead of merely seeking the global minimum of the loss function, the model optimizes for "flatness" in the loss landscape. This ensures that small perturbations in the input or parameters—often caused by noisy labels—do not lead to significant performance degradation, enhancing the model's generalization capabilities.
Experimental Results
The FIVE framework was evaluated on three benchmark datasets: 3D MNIST, ModelNet10, and ModelNet40. The results demonstrate a significant leap over existing state-of-the-art (SOTA) methods, particularly in low-label regimes.
On the ModelNet40 dataset with only 800 labeled samples, FIVE achieved a 72.94% Mean Average Precision (mAP) for 2D-to-3D retrieval, outperforming the previous best method (RONO) by a substantial 22.10%. Similar improvements were observed on 3D MNIST, where FIVE surpassed the next best method by an average of 4.74%.
Qualitative analysis using t-SNE visualizations confirmed that FIVE produces tighter, more distinct clusters where 2D and 3D embeddings are well-intermingled, indicating successful cross-modal alignment.
Conclusion
FIVE introduces a robust approach to semi-supervised 2D-3D retrieval by moving beyond simple class centroids to a more nuanced voting mechanism. By combining this with heterogeneous feature mixup and sharpness-aware optimization, the framework effectively bridges the modality gap with minimal supervision. Future work will explore extending this approach to zero-shot retrieval and more complex, multi-object 3D scenes.
Summary updated