Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognition

Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, Hanwang Zhang

Abstract

Recognizing 3D objects with minimal data remains a significant challenge in computer vision. While leveraging pre-trained 2D foundation models is a common strategy, simple fusion often fails due to "joint hard samples"—instances where both 2D and 3D modalities provide high-confidence but conflicting incorrect predictions. This paper introduces INVJOINT, an invariant training strategy that harmonizes 2D and 3D representations by focusing on these hard samples. By utilizing Invariant Risk Minimization (IRM) to extract features consistent across both domains, INVJOINT significantly reduces modality conflict. Experimental results demonstrate state-of-the-art performance, including a 15.89% improvement in 1-shot accuracy on the ModelNet40 benchmark.

Introduction: The Challenge of Few-Shot 3D Recognition

Deep learning models for 3D point clouds are notoriously data-hungry due to the sparse, irregular, and unstructured nature of the data. In few-shot scenarios, where only a handful of examples are available for new categories, standard 3D models struggle to generalize. A common workaround is to project 3D point clouds into 2D images to leverage powerful foundation models like CLIP. However, a "naive ensemble"—simply averaging predictions from 3D and 2D branches—often yields marginal benefits.

The authors identify joint hard samples as the primary obstacle. These are specific objects where both 2D and 3D models are confidently wrong in different ways, leading to a persistent modality conflict. Furthermore, models often fall into "shortcut traps," relying on features like rendering artifacts that do not generalize. INVJOINT addresses these issues by forcing the model to ignore modality-specific biases and focus on invariant features.

Methodology: Invariant Training and Hard Sample Selection

INVJOINT employs a dual-branch architecture consisting of a 3D branch (using DGCNN) and a 2D branch (using a differentiable renderer and a frozen CLIP encoder). The training process is iterative and centers on two key steps:

  1. Joint Hard Sample Selection: The system identifies hard samples by fitting a Gaussian Mixture Model (GMM) to the training loss. It specifically targets samples where the 2D and 3D models exhibit high discrepancy or high-confidence errors.
  2. Cross-modal Invariant Learning: Using Invariant Risk Minimization (IRM), the 2D and 3D features are treated as distinct "environments." A learnable gate function (a soft mask) filters out modality-specific biases, retaining only the features that are invariant across both domains.
Overview of the INVJOINT architecture and training process

Finally, a Cross-modality Alignment Loss ensures that the filtered 2D and 3D features of the same object are aligned in the feature space, reducing the conflict ratio and improving overall robustness.

Experimental Evaluation

The researchers evaluated INVJOINT across several challenging benchmarks, including ModelNet10/40, ScanObjectNN (real-world noisy scans), and Toys4K (a low-shot dataset with 105 categories).

Few-shot classification accuracy on ScanObjectNN, Toys4K, and ModelNet10

Key results include:

  • ModelNet40: Achieved 68.85% accuracy in the 1-shot setting, representing a 15.89% improvement over the previous state-of-the-art, PointCLIP.
  • Toys4K: Reached 93.52% accuracy in the 16-shot setting, significantly outperforming pure 3D models.
  • ScanObjectNN: Improved 16-shot accuracy from a 51.90% baseline to 57.02%, demonstrating robustness to real-world noise and occlusions.

Conclusion and Future Work

INVJOINT demonstrates that explicitly addressing modality conflicts through invariant training is a powerful approach for few-shot 3D recognition. By treating 2D models as teachers rather than just auxiliary inputs, INVJOINT extracts more robust features from sparse 3D data. Future work may extend this invariant training logic to denser tasks such as 3D part segmentation and object detection, or integrate it with evolving backbones like Point Transformers.

Summary updated

Abstract

We tackle the data scarcity challenge in few-shot point cloud recognition of 3D objects by using a joint prediction from a conventional 3D model and a well-trained 2D model. Surprisingly, such an ensemble, though seems trivial, has hardly been shown effective in recent 2D-3D models. We find out the crux is the less effective training for the ''joint hard samples'', which have high confidence prediction on different wrong labels, implying that the 2D and 3D models do not collaborate well. To this end, our proposed invariant training strategy, called InvJoint, does not only emphasize the training more on the hard samples, but also seeks the invariance between the conflicting 2D and 3D ambiguous predictions. InvJoint can learn more collaborative 2D and 3D representations for better ensemble. Extensive experiments on 3D shape classification with widely adopted ModelNet10/40, ScanObjectNN and Toys4K, and shape retrieval with ShapeNet-Core validate the superiority of our InvJoint.
BibTeX
@inproceedings{yi2023invariant,
  author = {Xuanyu Yi and Jiajun Deng and Qianru Sun and Xian-Sheng Hua and Joo-Hwee Lim and Hanwang Zhang},
  title = {{Invariant Training 2D-3D Joint Hard Samples for Few-Shot Point Cloud Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01330},
}
Figure