Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on Compressed Spatial Tokens

Shuqi Lu, Haowei Lin, Lin Yao, Zhifeng Gao, Xiaohong Ji, Weinan E, Linfeng Zhang, Guolin Ke

Abstract

Recent advancements in large language models and their multi-modal extensions have demonstrated the effectiveness of unifying generation and understanding through autoregressive next-token prediction. However, despite the critical role of 3D structural generation and understanding (3D GU) in AI for science, these tasks have largely evolved independently, with autoregressive methods remaining underexplored. To bridge this gap, we introduce Uni-3DAR, a unified framework that seamlessly integrates 3D GU tasks via autoregressive prediction. At its core, Uni-3DAR employs a novel hierarchical tokenization that compresses 3D space using an octree, leveraging the inherent sparsity of 3D structures. It then applies an additional tokenization for fine-grained structural details, capturing key attributes such as atom types and precise spatial coordinates in microscopic 3D structures. We further propose two optimizations to enhance efficiency and effectiveness. The first is a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. The second is a masked next-token prediction mechanism tailored for dynamically varying token positions, significantly boosting model performance. By combining these strategies, Uni-3DAR successfully unifies diverse 3D GU tasks within a single autoregressive framework. Extensive experiments across multiple microscopic 3D GU tasks, including molecules, proteins, polymers, and crystals, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256% relative improvement while delivering inference speeds up to 21.8x faster.

Introduction

Despite sharing fundamental spatial principles, 3D modeling has historically evolved in silos. A model trained to reconstruct a 3D chair is often incompatible with predicting protein folding, and crystal generators are rarely adaptable for molecular docking. This fragmentation stems from two primary challenges: scale/sparsity and dynamic topology. 3D data is inherently sparse, making dense representations inefficient, and the critical components of 3D structures vary wildly in position and density compared to the fixed grid of 2D images.

Current approaches frequently rely on Diffusion Models, which, while powerful, are computationally expensive and often require a fixed number of points or atoms to be defined upfront. Uni-3DAR addresses these issues by proposing a "GPT moment" for 3D modeling: a single autoregressive architecture that treats 3D structures as 1D token sequences.

The Uni-3DAR Framework

The core of Uni-3DAR is a sophisticated tokenization and prediction pipeline designed to handle the unique constraints of 3D space.

Coarse-to-Fine Octree Tokenizer

The model recursively subdivides 3D space into an octree. Empty regions are pruned, while occupied regions are subdivided until they reach a "leaf" level. These leaves are treated as 3D patches containing fine-grained details, such as atom types or VQ-VAE codes.

Hierarchical spatial subdivision levels from Level 5 (1x1x1) to Level 0 (fine-grained atoms).

2-Level Subtree Compression

To maintain manageable sequence lengths, Uni-3DAR groups eight subcells into a single 8-bit token, representing 256 possible occupancy states. This reduces the sequence length by approximately 8x, enabling the modeling of large structures like proteins that would otherwise exceed memory limits.

The Uni-3DAR tokenization pipeline showing octree construction and subtree compression.

Masked Next-Token Prediction (MNTP)

Standard autoregressive models struggle when the position of the next token is unknown. Uni-3DAR solves this by duplicating each token: it provides the model with the position and a [MASK] token, requiring the model to predict the content. This decouples "where something is" from "what it is," allowing the framework to handle the dynamic nature of sparse 3D grids effectively.

Experimental Evaluation

The researchers evaluated Uni-3DAR across a diverse range of benchmarks, including microscopic molecular datasets (QM9, GEOM-DRUG), crystal structures (Carbon-24, MP-20), and macroscopic objects (ShapeNet). The model also tackled biological tasks like protein pocket prediction and molecular docking.

Uni-3DAR demonstrates a rare combination of superior performance and increased efficiency:

  • Performance: In PXRD-guided crystal structure prediction, it achieved a 256% relative improvement over previous state-of-the-art models. In molecular generation (QM9), it reached 93.7% molecular stability, surpassing specialized models like UniGEM.
  • Speed: By avoiding the iterative steps required by diffusion models, Uni-3DAR is up to 21.8x faster during inference.
  • Versatility: It is the first autoregressive model to successfully unify tasks as diverse as 3D object generation and molecular docking within a single framework.

Discussion and Future Directions

While Uni-3DAR provides a unified framework, current model instances were trained separately for each benchmark to ensure fair comparison with specialized baselines. The ultimate goal is the development of a 3D Foundation Model trained on a heterogeneous mixture of all these datasets simultaneously. Such a model could potentially learn universal 3D priors, allowing spatial knowledge from macroscopic objects to inform microscopic structural predictions. Extending this framework to larger-scale environments, such as full 3D scenes, remains a promising frontier for future research.

Summary updated

Abstract

Recent advancements in large language models and their multi-modal extensions have demonstrated the effectiveness of unifying generation and understanding through autoregressive next-token prediction. However, despite the critical role of 3D structural generation and understanding (3D GU) in AI for science, these tasks have largely evolved independently, with autoregressive methods remaining underexplored. To bridge this gap, we introduce Uni-3DAR, a unified framework that seamlessly integrates 3D GU tasks via autoregressive prediction. At its core, Uni-3DAR employs a novel hierarchical tokenization that compresses 3D space using an octree, leveraging the inherent sparsity of 3D structures. It then applies an additional tokenization for fine-grained structural details, capturing key attributes such as atom types and precise spatial coordinates in microscopic 3D structures. We further propose two optimizations to enhance efficiency and effectiveness. The first is a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. The second is a masked next-token prediction mechanism tailored for dynamically varying token positions, significantly boosting model performance. By combining these strategies, Uni-3DAR successfully unifies diverse 3D GU tasks within a single autoregressive framework. Extensive experiments across multiple microscopic 3D GU tasks, including molecules, proteins, polymers, and crystals, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256\% relative improvement while delivering inference speeds up to 21.8x faster. The code is publicly available at https://github.com/dptech-corp/Uni-3DAR.
BibTeX
@inproceedings{lu2026uni,
  author = {Shuqi Lu and Haowei Lin and Lin Yao and Zhifeng Gao and Xiaohong Ji and Weinan E and Linfeng Zhang and Guolin Ke},
  title = {{Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on Compressed Spatial Tokens}},
  booktitle = {FM4Science @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=RQW73gLLMp},
}
Figure