Abstract
Point2Real is a training-free framework designed to bridge the significant domain gap between sparse 3D point clouds and 2D Vision-Language models (VLMs) like CLIP. By transforming discrete point clouds into realistically rendered meshes with plausible textures and identifying optimal viewpoints, Point2Real enables 2D models to achieve state-of-the-art 3D recognition accuracy in zero-shot and few-shot settings. Experimental results on benchmarks like ModelNet and ScanObjectNN demonstrate substantial improvements over existing projection-based methods, highlighting the importance of realistic visual representations for unlocking the latent knowledge within pre-trained 2D models for 3D understanding.
Introduction
The advancement of 2D image recognition, powered by large-scale VLMs trained on billions of internet images, has not been fully mirrored in the 3D domain due to the scarcity of annotated 3D data. A primary obstacle is the "domain gap" between unordered, sparse 3D point clouds and the natural images VLMs are trained on. Traditional methods often project point clouds into 2D depth maps, which typically lack texture and geometric continuity, preventing VLMs from reaching their full potential.
Methodology
Point2Real addresses the domain gap through a three-stage pipeline that transforms a "dot cloud" into a representation easily understood by 2D models:
- Shape Recovery Module: Instead of simple projection, the framework uses an Iterative Ball-Pivoting Algorithm (BPA). This process "skins" the points into a continuous mesh, filling holes and smoothing surfaces to restore a clear geometric structure.
- Texture Rendering Module: Recognizing that VLMs are highly sensitive to texture, Point2Real applies a set of candidate textures (e.g., wood, metal, leather) to the mesh. It uses ray-tracing to render realistic images and employs CLIP’s own confidence scores to select the most "natural" texture for a specific object.
- Multi-view Adapter: To identify the most informative viewpoints, the authors developed a heuristic that calculates the "depth surface area" of a view. A larger surface area serves as a proxy for higher geometric complexity, indicating a more discriminative viewpoint.
For few-shot learning scenarios, the framework introduces a Dual-path Network that fuses features from both the realistically rendered images and traditional depth maps, leveraging the strengths of both representations.
Key Results and Insights
The researchers evaluated Point2Real on standard 3D benchmarks including ModelNet10, ModelNet40, and the real-world ScanObjectNN dataset.
- Zero-Shot Performance: On ModelNet10, Point2Real achieved 79.5% accuracy, outperforming the previous state-of-the-art (ReCon) by 3.9%. On the challenging ScanObjectNN
OBJ_BGsplit, it outperformed PointCLIP V2 by a massive 11.8% (53.0% vs 41.2%). - Few-Shot Gains: In 1-shot scenarios on ModelNet40, the framework reached 67.5% accuracy, a significant jump over PointCLIP's 52.6%.
- Texture as a Feature: A key insight is that adding artificial textures significantly boosts recognition. Even without original color data, providing a plausible "skin" helps the 2D visual encoder trigger correct semantic associations.
- Geometry as Information Density: By mathematically identifying angles with the most "undulation" via depth surface area, the Multi-view Adapter selects optimal views without human labeling.
Conclusion
Point2Real demonstrates that a well-engineered "translation" layer from 3D to realistic 2D can outperform models trained directly on 3D data. While the current system faces computational overhead from high-quality rendering (approx. 3s per object in Blender), faster engines like PyTorch3D offer a viable path forward. Future work may involve using generative AI for dynamic, object-specific texture generation and refining viewpoint selection with semantic cues.
Summary updated