Toward a Unified Representation of Multi-Modal Pre-Training for 3-D Processing

Ben Fei, Yixuan Li, Weidong Yang, Lipeng Ma, Ying HeIEEE TVCG 2025 · DOI

Abstract

With the growing demand for real-world 3-D understanding, learning effective representations of 3-D data has become increasingly important for tasks such as shape classification, model retrieval, scene reconstruction, and point cloud completion. Although previous work has explored self-supervised learning within individual modalities (e.g., point clouds or images), the potential of multi-modal supervision remains largely underexplored due to the lack of aligned and scalable training signals. In this work, we present DR-Point, a tri-modal pre-training framework that jointly learns from RGB images, depth maps, and 3-D point clouds to build a unified embedding space across modalities. By enforcing cross-modal consistency among RGB-depth-point triplets, DR-Point achieves effective 2-D-3-D feature alignment without manual annotations. A differentiable rendering module further enhances geometric fidelity by synthesizing depth cues and refining structural details in reconstructed point clouds. Extensive experiments on benchmarks demonstrate that DR-Point consistently outperforms state-of-the-art self-supervised methods on 3-D classification, segmentation, and completion. These results highlight the advantages of multi-modal pre-training for unified 3-D understanding and its potential to benefit a wide range of vision and graphics applications.
BibTeX
@article{fei2025toward,
  author = {Ben Fei and Yixuan Li and Weidong Yang and Lipeng Ma and Ying He},
  title = {{Toward a Unified Representation of Multi-Modal Pre-Training for 3-D Processing}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2025},
  doi = {10.1109/tvcg.2025.3631434},
}

Loading related content...

Figure