Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition

Hanxuan Li, Bin Fu, Ruiping Wang, Xilin Chen

Abstract

Point2Real is a training-free framework designed to bridge the significant domain gap between sparse 3D point clouds and 2D Vision-Language models (VLMs) like CLIP. By transforming discrete point clouds into realistically rendered meshes with plausible textures and identifying optimal viewpoints, Point2Real enables 2D models to achieve state-of-the-art 3D recognition accuracy in zero-shot and few-shot settings. Experimental results on benchmarks like ModelNet and ScanObjectNN demonstrate substantial improvements over existing projection-based methods, highlighting the importance of realistic visual representations for unlocking the latent knowledge within pre-trained 2D models for 3D understanding.

Introduction

The advancement of 2D image recognition, powered by large-scale VLMs trained on billions of internet images, has not been fully mirrored in the 3D domain due to the scarcity of annotated 3D data. A primary obstacle is the "domain gap" between unordered, sparse 3D point clouds and the natural images VLMs are trained on. Traditional methods often project point clouds into 2D depth maps, which typically lack texture and geometric continuity, preventing VLMs from reaching their full potential.

Point2Real bridges the domain gap by converting point clouds into realistic images, significantly reducing the distance to the target vision-language domain compared to simple projection.

Methodology

Point2Real addresses the domain gap through a three-stage pipeline that transforms a "dot cloud" into a representation easily understood by 2D models:

  1. Shape Recovery Module: Instead of simple projection, the framework uses an Iterative Ball-Pivoting Algorithm (BPA). This process "skins" the points into a continuous mesh, filling holes and smoothing surfaces to restore a clear geometric structure.
  2. Texture Rendering Module: Recognizing that VLMs are highly sensitive to texture, Point2Real applies a set of candidate textures (e.g., wood, metal, leather) to the mesh. It uses ray-tracing to render realistic images and employs CLIP’s own confidence scores to select the most "natural" texture for a specific object.
  3. Multi-view Adapter: To identify the most informative viewpoints, the authors developed a heuristic that calculates the "depth surface area" of a view. A larger surface area serves as a proxy for higher geometric complexity, indicating a more discriminative viewpoint.
The Point2Real framework includes modules for shape recovery, texture rendering, and multi-view selection, integrated with LLM-assisted prompting for downstream tasks.

For few-shot learning scenarios, the framework introduces a Dual-path Network that fuses features from both the realistically rendered images and traditional depth maps, leveraging the strengths of both representations.

Key Results and Insights

The researchers evaluated Point2Real on standard 3D benchmarks including ModelNet10, ModelNet40, and the real-world ScanObjectNN dataset.

  • Zero-Shot Performance: On ModelNet10, Point2Real achieved 79.5% accuracy, outperforming the previous state-of-the-art (ReCon) by 3.9%. On the challenging ScanObjectNN OBJ_BG split, it outperformed PointCLIP V2 by a massive 11.8% (53.0% vs 41.2%).
  • Few-Shot Gains: In 1-shot scenarios on ModelNet40, the framework reached 67.5% accuracy, a significant jump over PointCLIP's 52.6%.
  • Texture as a Feature: A key insight is that adding artificial textures significantly boosts recognition. Even without original color data, providing a plausible "skin" helps the 2D visual encoder trigger correct semantic associations.
  • Geometry as Information Density: By mathematically identifying angles with the most "undulation" via depth surface area, the Multi-view Adapter selects optimal views without human labeling.

Conclusion

Point2Real demonstrates that a well-engineered "translation" layer from 3D to realistic 2D can outperform models trained directly on 3D data. While the current system faces computational overhead from high-quality rendering (approx. 3s per object in Blender), faster engines like PyTorch3D offer a viable path forward. Future work may involve using generative AI for dynamic, object-specific texture generation and refining viewpoint selection with semantic cues.

Summary updated

Abstract

Recognition in open-world scenarios is an important and challenging field, where Vision-Language Pre-training paradigms have greatly impacted the 2D domain. This inspires a growing interest in introducing 2D pre-trained models, such as CLIP, into the 3D domain to enhance the ability of point cloud understanding. Considering the difference between discrete 3D point clouds and real-world 2D images, reducing the domain gap is crucial. Some recent works project point clouds onto a 2D plane to enable 3D zero-shot capabilities without training. However, this simplistic approach leads to an unclear or even distorted geometric structure, limiting the potential of 2D pre-trained models in 3D. To address the domain gap, we propose Point2Real, a training-free framework based on the realistic rendering technique to automate the transformation of the 3D point cloud domain into the Vision-Language domain. Specifically, Point2Real leverages a shape recovery module that devises an iterative ball-pivoting algorithm to convert point clouds into meshes, narrowing the gap in shape at first. To simulate photo-realistic images, a set of refined textures as candidates is applied for rendering, where the CLIP confidence is utilized to select the suitable one. Moreover, to tackle the viewpoint challenge, a heuristic multi-view adapter is implemented for feature aggregation, which exploits the depth surface as an effective indicator of view-specific discriminability for recognition. We conduct experiments on ModelNet10, ModelNet40, and ScanObjectNN datasets, and the results demonstrate that Point2Real outperforms other approaches in zero-shot and few-shot tasks by a large margin.
BibTeX
@inproceedings{li2024point,
  author = {Hanxuan Li and Bin Fu and Ruiping Wang and Xilin Chen},
  title = {{Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28088},
}
Figure