Abstract
Few-shot 3D point cloud recognition remains a significant challenge due to data scarcity and the inherent sparsity of 3D representations. While integrating 2D vision models (e.g., CLIP) offers a potential solution, naive ensembles often fail because 2D and 3D modalities exhibit conflicting, confident errors on "joint hard samples." This paper introduces INVJOINT, a training strategy that leverages Invariant Risk Minimization (IRM) to force collaboration between 2D and 3D models. By identifying joint hard samples via a Gaussian Mixture Model and filtering out modality-specific biases, INVJOINT achieves state-of-the-art results, including a 15.89% improvement in 1-shot accuracy on ModelNet40.
Introduction
Recognizing 3D objects from sparse point clouds is notoriously difficult, especially in few-shot learning scenarios where labeled data is minimal. While 2D computer vision has benefited from massive pre-trained models like CLIP, 3D models often struggle with data scarcity. A common-sense solution is to combine the two: project the 3D point cloud into 2D images and use a joint prediction from both a 3D and a 2D model.
However, research indicates that a "naive ensemble"—simply averaging the results of both modalities—barely improves performance. The core issue lies in "joint hard samples": instances where both the 2D and 3D models are confidently wrong, often about different labels. When both models are confidently incorrect, the ensemble remains confused. INVJOINT addresses this by forcing 2D and 3D representations to become more collaborative through invariant training.
Methodology
The INVJOINT framework consists of two main branches: a 3D branch using a standard Dynamic Graph CNN (DGCNN) to process raw point clouds, and a 2D branch that uses a differentiable renderer to convert point clouds into multi-view images, which are then processed by a frozen CLIP visual encoder with a lightweight adapter.
The core of INVJOINT is a two-step iterative training process designed to resolve modality conflicts:
- Joint Hard Sample Selection: The system employs a Gaussian Mixture Model (GMM) to analyze the loss distribution across the dataset. This allows the model to identify samples where the 2D and 3D models have high confidence in conflicting, incorrect labels.
- Cross-modal Invariant Learning: Inspired by Invariant Risk Minimization (IRM), the researchers treat the 2D and 3D modalities as two different "environments." They train a learnable "gate function" (a soft mask) that filters out modality-specific biases—such as rendering artifacts in 2D or sparsity issues in 3D. This process ensures the model retains only the features that are invariant across both domains, leading to a more robust shared understanding of the object.
Key Insights
The "Shortcut" Bias
Models often take "shortcuts" by relying on modality-specific features. For instance, a 2D model might rely on a specific texture or silhouette artifact that does not exist in the actual 3D geometry. INVJOINT forces the model to ignore these shortcuts by focusing on features that persist across both 2D and 3D views.
Conflict as a Signal
Instead of attempting to fix every error indiscriminately, INVJOINT specifically targets samples where the modalities disagree. By resolving these conflicts during training, the models learn to reconcile their differences and form a more cohesive representation of the underlying data.
Ensemble-First Training
While many 2D-3D methods use 2D knowledge to "distill" a 3D model and then discard the 2D component at inference, INVJOINT argues that maintaining both and training them to be collaborative is superior for few-shot scenarios.
Experimental Results
INVJOINT was evaluated across several diverse benchmarks, significantly outperforming existing state-of-the-art (SOTA) methods:
- ModelNet40: The model achieved 68.85% accuracy in a 1-shot setting, representing an absolute improvement of 15.89% over PointCLIP.
- Toys4K: In the 16-shot setting, it reached 93.52% accuracy, demonstrating an ability to generalize across a large number of categories (105) where 3D-only models typically struggle.
- ScanObjectNN: On this challenging real-world dataset containing scanned objects with noise and occlusions, INVJOINT maintained a consistent lead, proving that its invariant features are robust to real-world scanning artifacts.
- ShapeNet-Core: The method achieved 84.1% mAP in 3D shape retrieval, setting a new benchmark for retrieval using only 10 rendered views.
Discussion and Limitations
While INVJOINT provides a significant performance boost, it introduces certain trade-offs. The use of a differentiable renderer and multi-view processing increases computational complexity compared to 3D-only models. Additionally, the performance is heavily dependent on the quality of the 2D backbone (CLIP); if the 2D model possesses inherent blind spots for specific categories, the ensemble's performance may be capped.
Future research may extend this invariant training logic to other 3D tasks beyond classification, such as part segmentation and object detection in complex, cluttered scenes.
Summary updated