Abstract
Recognizing 3D objects with minimal data remains a significant challenge in computer vision. While leveraging pre-trained 2D foundation models is a common strategy, simple fusion often fails due to "joint hard samples"—instances where both 2D and 3D modalities provide high-confidence but conflicting incorrect predictions. This paper introduces INVJOINT, an invariant training strategy that harmonizes 2D and 3D representations by focusing on these hard samples. By utilizing Invariant Risk Minimization (IRM) to extract features consistent across both domains, INVJOINT significantly reduces modality conflict. Experimental results demonstrate state-of-the-art performance, including a 15.89% improvement in 1-shot accuracy on the ModelNet40 benchmark.
Introduction: The Challenge of Few-Shot 3D Recognition
Deep learning models for 3D point clouds are notoriously data-hungry due to the sparse, irregular, and unstructured nature of the data. In few-shot scenarios, where only a handful of examples are available for new categories, standard 3D models struggle to generalize. A common workaround is to project 3D point clouds into 2D images to leverage powerful foundation models like CLIP. However, a "naive ensemble"—simply averaging predictions from 3D and 2D branches—often yields marginal benefits.
The authors identify joint hard samples as the primary obstacle. These are specific objects where both 2D and 3D models are confidently wrong in different ways, leading to a persistent modality conflict. Furthermore, models often fall into "shortcut traps," relying on features like rendering artifacts that do not generalize. INVJOINT addresses these issues by forcing the model to ignore modality-specific biases and focus on invariant features.
Methodology: Invariant Training and Hard Sample Selection
INVJOINT employs a dual-branch architecture consisting of a 3D branch (using DGCNN) and a 2D branch (using a differentiable renderer and a frozen CLIP encoder). The training process is iterative and centers on two key steps:
- Joint Hard Sample Selection: The system identifies hard samples by fitting a Gaussian Mixture Model (GMM) to the training loss. It specifically targets samples where the 2D and 3D models exhibit high discrepancy or high-confidence errors.
- Cross-modal Invariant Learning: Using Invariant Risk Minimization (IRM), the 2D and 3D features are treated as distinct "environments." A learnable gate function (a soft mask) filters out modality-specific biases, retaining only the features that are invariant across both domains.
Finally, a Cross-modality Alignment Loss ensures that the filtered 2D and 3D features of the same object are aligned in the feature space, reducing the conflict ratio and improving overall robustness.
Experimental Evaluation
The researchers evaluated INVJOINT across several challenging benchmarks, including ModelNet10/40, ScanObjectNN (real-world noisy scans), and Toys4K (a low-shot dataset with 105 categories).
Key results include:
- ModelNet40: Achieved 68.85% accuracy in the 1-shot setting, representing a 15.89% improvement over the previous state-of-the-art, PointCLIP.
- Toys4K: Reached 93.52% accuracy in the 16-shot setting, significantly outperforming pure 3D models.
- ScanObjectNN: Improved 16-shot accuracy from a 51.90% baseline to 57.02%, demonstrating robustness to real-world noise and occlusions.
Conclusion and Future Work
INVJOINT demonstrates that explicitly addressing modality conflicts through invariant training is a powerful approach for few-shot 3D recognition. By treating 2D models as teachers rather than just auxiliary inputs, INVJOINT extracts more robust features from sparse 3D data. Future work may extend this invariant training logic to denser tasks such as 3D part segmentation and object detection, or integrate it with evolving backbones like Point Transformers.
Summary updated