When ChatGPT for Computer Vision Will Come? From 2D to 3D

Chenghao Li, Chaoning ZhangarXiv 2023 · DOI · arXiv

Abstract

While Large Language Models (LLMs) have revolutionized natural language processing (NLP) and 2D computer vision is seeing the rise of general-task models like Segment Anything (SAM), 3D vision remains in a fragmented, task-specific state. This paper explores the evolutionary stages of AI models and identifies data scarcity as the primary bottleneck for 3D foundation models. The authors argue that a "ChatGPT moment" for 3D vision requires a consumer-level hardware breakthrough to democratize 3D data capture, shifting the field from professional content to abundant user-generated content.

Introduction

The AI world has been transformed by Large Language Models like GPT-4, which can solve nearly any text-based task with a single architecture. While 2D computer vision is making significant strides toward this level of universality, the 3D vision field remains fragmented and lagging. This perspective paper investigates the fundamental question: What is preventing the emergence of a "foundation model" for 3D vision? The authors analyze this gap through the lenses of model architecture evolution and the shifting landscape of data production.

The Three-Stage Evolution of AI Models

The authors propose a conceptual framework to categorize the maturity of different AI modalities into three distinct stages:

  1. Task-specific stage: Characterized by small datasets and models with strong inductive biases (e.g., CNNs for images, RNNs for text, VoxNet for 3D).
  2. Fine-tuning stage: Involves large-scale pre-training on diverse data followed by specific downstream tuning (e.g., BERT for text, Vision Transformers for 2D, Point-BERT for 3D).
  3. General-task stage: Massive models that perform zero-shot tasks via prompting (e.g., the GPT series for text, SAM for 2D).
The evolution of AI models across NLP, 2D, and 3D vision through three developmental stages.

As illustrated above, while text has fully transitioned to the general-task stage and 2D vision is in a transition phase, 3D vision is still largely stuck in the task-specific or early fine-tuning stages. We have "Point-BERT," but we do not yet have a "3D-GPT."

The Data Production Pipeline: PGC, UGC, and AIGC

A critical insight of this research is the dependency of AI progress on the volume and type of available data. The authors trace the shift from PGC (Professionally Generated Content) to UGC (User Generated Content), and finally to AIGC (AI Generated Content).

The relationship between Professionally Generated Content (PGC), User Generated Content (UGC), and AI Generated Content (AIGC).

They argue that the success of AIGC is essentially parasitic on the volume of PGC and UGC available for training. In the 3D domain, data remains "complex, scarce, and diversified." Most 3D data is still expensive PGC, lacking the massive volume of UGC that fueled the 2D and text revolutions.

The authors posit that the 2D AI revolution was not just a software breakthrough, but a hardware one. The release of the iPhone 4 and the subsequent rise of social media made high-quality image capture and sharing (UGC) frictionless. This provided the "big data" required for deep learning to flourish.

For 3D AIGC to reach its full potential, the world needs a consumer device that makes 3D scanning as easy as taking a photo. This "3D iPhone 4" would transition 3D data from expensive, professional production to abundant, user-generated content. Without this democratization of data capture, 3D models will continue to face a "representation problem" where computational complexity and lack of a unified format prevent scaling to the size of LLMs.

Conclusion

The arrival of a "ChatGPT for Computer Vision" in the 3D realm is inevitable, but it is tied more to the democratization of 3D data capture than to architectural tweaks alone. Future progress must focus on improving the usability of 3D capture tools and determining if 3D models will follow the same scaling laws as LLMs once sufficient data is available. The field currently awaits a leading enterprise to define the 3D ecosystem and provide the hardware-software integration necessary for a data explosion.

Summary updated

Abstract

ChatGPT and its improved variant GPT4 have revolutionized the NLP field with a single model solving almost all text related tasks. However, such a model for computer vision does not exist, especially for 3D vision. This article first provides a brief view on the progress of deep learning in text, image and 3D fields from the model perspective. Moreover, this work further discusses how AIGC evolves from the data perspective. On top of that, this work presents an outlook on the development of AIGC in 3D from the data perspective.
BibTeX
@misc{li2023when,
  author = {Chenghao Li and Chaoning Zhang},
  title = {{When ChatGPT for Computer Vision Will Come? From 2D to 3D}},
  year = {2023},
  eprint = {2305.06133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.06133},
}
Figure