Transformed Multi-view 3D Shape Features with Contrastive Learning

Márcus Vinícius Lobo Costa, Sherlon Almeida da Silva, Barbara Caroline Benato, Leo Sampaio Ferraz Ribeiro, Moacir Antonelli PontiarXiv 2025 · DOI · arXiv

Abstract

Understanding 3D shapes from 2D projections remains a significant challenge in computer vision. This paper investigates the synergy between Vision Transformers (ViTs) and modern contrastive learning objectives for 3D shape understanding. By processing multi-view 2D snapshots of 3D objects, the proposed approach demonstrates that ViTs significantly outperform traditional Convolutional Neural Networks (CNNs) in capturing global geometric relationships. Our method achieves state-of-the-art results on the ModelNet10 benchmark, reaching 90.6% classification accuracy and 95.5% mean Average Precision in retrieval tasks.

Introduction

3D shape understanding is a fundamental requirement for applications in robotics, autonomous navigation, and virtual reality. Historically, the dominant paradigm for processing 3D data involves multi-view analysis, where a series of 2D "snapshots" of a 3D object are processed by Convolutional Neural Networks (CNNs). While effective, CNNs possess inherent limitations: they often struggle to model long-range dependencies between disparate views and typically necessitate extensive manually labeled datasets for supervised training.

This research explores whether Vision Transformers (ViTs) and Contrastive Learning can overcome these hurdles. ViTs, through their self-attention mechanism, are naturally suited for capturing global context across multiple views. Simultaneously, contrastive learning provides a robust framework for learning discriminative features by maximizing similarity between related views while minimizing it for unrelated ones, effectively utilizing both labeled and unlabeled data.

Methodology

The proposed framework operates through a two-stage pipeline designed to extract and refine 3D shape features from 2D image sequences.

Multi-view Acquisition and Pre-training

The first stage involves rendering 12 consecutive 2D views of 3D meshes from the ModelNet10 and ModelNet40 datasets. These views are generated using the VEDO library to ensure consistent perspective and coverage. These images are then processed by an encoder backbone. The study rigorously compares the performance of the classic ResNet-50 CNN against various ViT architectures, including Small (S), Base (B), and DINO-pretrained variants.

Figure 1: The two-stage pipeline for multi-view acquisition, contrastive pre-training, and downstream task evaluation.

Contrastive Learning Objectives

The backbones are trained using five distinct contrastive loss functions to determine the most effective objective for shape discrimination:

  • Self-supervised: InfoNCE and SimCLR, which learn representations without requiring explicit labels.
  • Supervised: SupCon, ε-SupInfoNCE, and SINCERE. These objectives utilize class labels to "pull" embeddings of the same category together while "pushing" different categories apart in the feature space.

Following pre-training, the backbones are frozen, and their feature representations are evaluated on downstream tasks: 3D shape classification (via linear probing and k-NN) and 3D shape retrieval.

Key Insights and Discussion

The experimental results yield several critical observations regarding the architecture and optimization of 3D shape descriptors.

The Superiority of Vision Transformers

A primary finding is the consistent dominance of ViT architectures over CNNs. For example, the DINO-S/16 variant achieved an 81.87% Top-1 accuracy on ModelNet10, outperforming the ResNet-50 baseline (37.57%) by over 44 percentage points. This suggests that the attention mechanism in ViTs is significantly more capable of synthesizing a coherent 3D concept from a sequence of 2D views than the local receptive fields of standard convolutions.

Supervised Contrastive Learning

While self-supervised methods are valuable for leveraging unlabeled data, supervised contrastive losses—particularly SINCERE and SupCon—consistently produced the most discriminative features. These methods significantly outperformed standard Cross-Entropy baselines, indicating that the contrastive framework is better at refining local discriminative features while the ViT backbone captures global semantics.

Semantic Overlap

Despite the high performance, qualitative analysis indicates that models still struggle with fine-grained geometric differentiation. Structurally similar categories, such as "desks" and "tables," often exhibit significant overlap in the embedding space. This suggests that while global shape is well-captured, extremely subtle geometric nuances remain a challenge for current architectures.

Key Results

The proposed approach sets new benchmarks for multi-view 3D shape understanding. On the ModelNet10 dataset, the combination of a ViT-B/16 backbone with the SINCERE loss achieved a state-of-the-art 90.6% classification accuracy. In retrieval tasks, this configuration reached a 95.5% mean Average Precision (mAP).

Figure 6: Comparison of the proposed method against existing state-of-the-art multi-view 3D shape understanding models on ModelNet10 and ModelNet40.

Furthermore, the learned features proved highly robust; on the more complex ModelNet40 dataset, the DINO-S/16 model with ε-SupInfoNCE achieved 87.0% accuracy using a simple k-Nearest Neighbor (k-NN) classifier, demonstrating the high representational quality of the embeddings even without specialized classification heads.

Conclusion and Future Work

This research demonstrates that the integration of Vision Transformers and supervised contrastive learning significantly advances the state-of-the-art in 3D shape understanding from multi-view data. While powerful, limitations remain in capturing the finest geometric details. Future research will focus on multi-scale architectures to capture both global and local nuances simultaneously, the development of hierarchical contrastive objectives to handle semantic relationships, and testing the robustness of these models on noisy, real-world LiDAR data.

Summary updated

BibTeX
@misc{costa2025transformed,
  author = {Márcus Vinícius Lobo Costa and Sherlon Almeida da Silva and Barbara Caroline Benato and Leo Sampaio Ferraz Ribeiro and Moacir Antonelli Ponti},
  title = {{Transformed Multi-view 3D Shape Features with Contrastive Learning}},
  year = {2025},
  eprint = {2510.19955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.19955},
}
Figure