G3PT: Unleash the Power of Autoregressive Modeling in 3D Generation via Cross-Scale Querying Transformer

Jinzhi Zhang, Feng Xiong, Guangyu Wang, Mu XuIJCAI 2025 · DOI

TL;DR

This paper introduces G3PT, a novel autoregressive model for 3D shape generation that overcomes the limitations of applying sequential models to unordered 3D data. By using a Cross-scale Querying Transformer (CQT) to map 3D point clouds into discrete tokens at various detail levels, G3PT enables effective coarse-to-fine generation and achieves state-of-the-art results on image-to-3D and text-to-3D tasks, while also demonstrating power-law scaling behaviors.

What Problem Does This Paper Address?

Generating high-quality 3D shapes has been a significant challenge, especially when trying to leverage the power of autoregressive (AR) models, which have revolutionized language and image generation. AR models excel at sequential data by predicting the "next token," but 3D data, particularly point clouds, is inherently unordered. Previous attempts to adapt AR models to 3D either imposed artificial ordering, leading to limitations in representing complex geometry, or struggled with scalability and training time. This paper aims to bridge this gap by developing a 3D-native AR model that can effectively handle unordered 3D data and scale efficiently.

Methods

G3PT employs a two-stage approach:

  1. Cross-scale Vector Quantization (CVQ): This stage converts unordered 3D point cloud data into discrete tokens at different levels of detail. The core innovation here is the Cross-scale Querying Transformer (CQT). Unlike traditional methods that rely on average pooling or bilinear interpolation (which require ordered data), CQT uses learnable "downsample" and "upsample" queries with cross-attention mechanisms. This allows it to globally connect tokens across different scales without needing a specific token order, effectively creating a hierarchical, level-of-detail representation.
  2. Cross-scale AutoRegressive Modeling (CAR): This stage leverages the tokens generated by CVQ to perform autoregressive generation. It reuses the CQT to align token dimensions across scales and then employs a causal transformer for "next-scale prediction." This means the model predicts tokens for a finer level of detail conditioned on tokens from coarser levels, enabling a progressive generation process from coarse to fine. G3PT also supports conditional generation, integrating image or text inputs through attention mechanisms.

Datasets

The paper primarily uses the Objaverse dataset for training and evaluation. This dataset is a large collection of annotated 3D objects.

Key Insights and Ideas

  • Unordered 3D data can be effectively tokenized hierarchically: The key insight is that 3D data naturally has level-of-detail characteristics. By mapping this to discrete tokens at different scales using CQT, G3PT creates a sequential relationship suitable for AR models without forcing an artificial order.
  • Cross-scale Querying Transformer (CQT) is crucial: This novel component allows for global information integration across different detail levels of 3D data, overcoming the limitations of sequential processing on unordered inputs.
  • Power-law scaling in 3D generation: For the first time in 3D generation, G3PT demonstrates distinct power-law scaling behaviors, suggesting that its performance improves predictably with increased model size, similar to large language models.

Key Results

  • State-of-the-art performance: G3PT significantly outperforms previous Large Reconstruction Models (LRMs) and diffusion-based 3D generation methods in terms of generation quality, as measured by IoU, Chamfer distance, and F-score. The 1.5B parameter model achieves an IoU of 87.6%, Chamfer distance of 0.013, and F-score of 83.0%.
  • Versatile conditional generation: G3PT demonstrates strong performance in both image-to-3D and text-to-3D generation tasks, producing high-quality, semantically aligned, and diverse 3D shapes.
  • Efficient tokenization: Ablation studies show that G3PT's CVQ method outperforms VAE and VQVAE tokenizers, achieving near-complete codebook usage and better quantitative metrics.

Limitations and Future Work

  • Conditional generation exploration: While G3PT supports image and text conditioning, the paper notes that this is a preliminary attempt. Other advanced conditioning mechanisms are left for future exploration.
  • Training stability: The paper mentions using progressive training strategies for CAR to improve stability and accelerate convergence, implying that training large-scale 3D models can still be challenging.
  • Scalability: The paper highlights the power-law scaling, suggesting future work could focus on further scaling up the model to explore its full potential.

Summary updated

BibTeX
@inproceedings{zhang2025g,
  author = {Jinzhi Zhang and Feng Xiong and Guangyu Wang and Mu Xu},
  title = {{G3PT: Unleash the Power of Autoregressive Modeling in 3D Generation via Cross-Scale Querying Transformer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/262},
}
Figure