Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling

Shuqi Lu, Haowei Lin, Lin Yao, Zhifeng Gao, Xiaohong Ji, Yitao Liang, Weinan E, Linfeng Zhang, Guolin Ke

TL;DR

Uni-3DAR introduces a unified autoregressive framework that bridges the gap between macroscopic 3D object modeling and microscopic molecular/crystal structures. By compressing 3D space into discrete 1D token sequences using an optimized octree structure and a novel masked next-token prediction strategy, it consistently outperforms state-of-the-art diffusion models while delivering up to 21.8x faster inference speeds.

What Problem Does This Paper Address?

3D structure modeling is critical for a massive range of applications, from macroscopic computer vision (3D shape reconstruction) to microscopic biochemistry (protein folding, molecular docking, and crystal design). Despite sharing fundamental spatial principles, these domains have historically evolved in silos.

A model designed for macroscopic objects cannot easily adapt to the angstrom-scale sparsity of a molecule, and even models within the same scale rarely generalize across different tasks. This fragmentation prevents data reuse and leads to highly specialized, redundant architectures. The authors of this paper tackle this head-on by proposing a single, unified architecture capable of handling both 3D generation and 3D understanding across all scales.

Methods

At the core of Uni-3DAR is a sequence of technical innovations designed to turn sparse 3D data into compact, predictable 1D token sequences for a standard Transformer decoder:

  1. Coarse-to-Fine Octree Tokenizer: Instead of using a computationally heavy, dense voxel grid, the framework recursively subdivides 3D space into an octree structure. Branches corresponding to empty space are pruned, which naturally adapts to the extreme sparsity of molecular and surface data.
  2. Two-Level Subtree Compression (2LSC): To prevent the token sequence from exploding in length, the authors compress each two-level subtree (eight subcells) into a single 8-bit token. This reduces the sequence length by approximately 8x and converts eight binary classification steps into a single 256-class prediction task.
  3. Fine-Grained 3D Patches: For the final leaf nodes, the model encodes local sub-voxel details—such as exact atomic types and continuous coordinates for molecules, or Vector Quantized (VQ-VAE) tokens for macroscopic shapes.
  4. Masked Next-Token Prediction (MNTP): Because empty spaces are pruned dynamically, adjacent tokens no longer map to uniform intervals in 3D space. Standard autoregressive models struggle to predict the "next" token without knowing its target coordinates. MNTP solves this by duplicating each token: the first copy is replaced with a [MASK] token but retains its correct positional embedding. The model then predicts the content exclusively at this masked position, ensuring it is always conditioned on the correct target coordinates.
Overview of the Uni-3DAR framework showing the tokenization pipeline and multi-task capabilities.

Datasets

The model was evaluated across an extensive suite of benchmarks spanning multiple scales:

  • Small Molecules: QM9 (small molecules up to 29 atoms) and GEOM-DRUG (larger, complex drug-like compounds up to 181 atoms).
  • Crystals: Carbon-24, MP-20, and the highly challenging MPTS-52 dataset.
  • Macroscopic 3D Objects: ShapeNet (specifically the airplane, chair, and car categories).
  • Proteins & Bio-modeling: CASF-2016 core set, PDBBind v2020 refined set, and MOAD for protein pocket prediction and molecular docking tasks.
  • Polymers: Eight density functional theory (DFT)-calculated homopolymer property datasets.

Key Insights and Ideas

  • Autoregressive Models Can Outperform Diffusion in 3D: While diffusion models have recently dominated 3D molecular and shape generation, Uni-3DAR demonstrates that discrete tokenization paired with an autoregressive Transformer can actually yield superior structural validity and accuracy.
  • Decoupling Position from Content is Essential: Trying to predict where a token is and what it is simultaneously introduces too much noise when dealing with sparse trees. By providing the target position via a masked placeholder, the autoregressive generation becomes highly stable.
  • Generative and Understanding Tasks are Synergistic: The authors discovered that training the model on both generation and understanding tasks creates a powerful feedback loop. Incorporating a structure-level property prediction task improves the chemical validity of generated molecules, while adding an auxiliary generation loss enhances property prediction accuracy.
Detailed look at the coarse-to-fine subdivision and subtree compression mechanics.

Key Results

  • Massive Speedups: Uni-3DAR achieves inference speeds up to 21.8x faster than prior diffusion-based models like GeoLDM at large batch sizes.
  • State-of-the-Art Crystal Modeling: In Powder X-ray Diffraction (PXRD)-guided crystal structure prediction, Uni-3DAR cut the Root Mean Squared Error (RMSE) from 0.0707 down to 0.0276—a massive 256% relative improvement.
  • Superior Molecular Generation: On the QM9 benchmark, the model achieved a Molecule Stability of 93.7% and a Validity score of 98.0%, substantially outperforming specialized baselines.
  • Effective Molecular Docking: Operating solely on raw atom types and coordinates without complex feature engineering, it surpassed specialized models like SurfDock on Top-1 success rates for predicting how ligands bind to proteins.

Limitations and Future Work

The primary limitation acknowledged by the authors is that they have not yet trained a single, massive foundation model on a heterogeneous mixture of all tasks simultaneously. For fair comparison against existing baselines, separate model instances were trained for each specific benchmark. The authors note that realizing a true "GPT-2 moment" for cross-scale 3D modeling via large-scale joint pretraining remains their primary goal for future work.

Summary updated

Abstract

3D structure modeling is essential across scales, enabling applications from fluid simulation and 3D reconstruction to protein folding and molecular docking. Yet, despite shared 3D spatial patterns, current approaches remain fragmented, with models narrowly specialized for specific domains and unable to generalize across tasks or scales. We propose Uni-3DAR, a unified autoregressive framework for cross-scale 3D generation and understanding. At its core is a coarse-to-fine tokenizer based on octree data structures, which compresses diverse 3D structures into compact 1D token sequences. We further propose a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. To address the challenge of dynamically varying token positions introduced by compression, we introduce a masked next-token prediction strategy that ensures accurate positional modeling, significantly boosting model performance. Extensive experiments across multiple 3D generation and understanding tasks, including small molecules, proteins, polymers, crystals, and macroscopic 3D objects, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256\% relative improvement while delivering inference speeds up to 21.8x faster.
BibTeX
@inproceedings{lu2026unified,
  author = {Shuqi Lu and Haowei Lin and Lin Yao and Zhifeng Gao and Xiaohong Ji and Yitao Liang and Weinan E and Linfeng Zhang and Guolin Ke},
  title = {{Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling}},
  booktitle = {FM4Science @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=RQW73gLLMp},
}
Figure