Abstract
3D structure modeling is essential across scales, enabling applications from fluid simulation and 3D reconstruction to protein folding and molecular docking. Yet, despite shared 3D spatial patterns, current approaches remain fragmented, with models narrowly specialized for specific domains and unable to generalize across tasks or scales. We propose Uni-3DAR, a unified autoregressive framework for cross-scale 3D generation and understanding. At its core is a coarse-to-fine tokenizer based on octree data structures, which compresses diverse 3D structures into compact 1D token sequences. We further propose a two-level subtree compression strategy, which reduces the octree token sequence by up to 8x. To address the challenge of dynamically varying token positions introduced by compression, we introduce a masked next-token prediction strategy that ensures accurate positional modeling, significantly boosting model performance. Extensive experiments across multiple 3D generation and understanding tasks, including small molecules, proteins, polymers, crystals, and macroscopic 3D objects, validate its effectiveness and versatility. Notably, Uni-3DAR surpasses previous state-of-the-art diffusion models by a substantial margin, achieving up to 256% relative improvement while delivering inference speeds up to 21.8x faster.
Introduction: The Fragmentation of 3D Modeling
Despite sharing fundamental spatial principles, 3D modeling has historically evolved in silos. A model designed to generate crystals cannot be easily adapted to predict protein folding or reconstruct a 3D chair. This fragmentation prevents data reuse and leads to highly specialized, redundant architectures.
The technical challenge lies in the nature of 3D data itself: it is incredibly sparse. While 2D images have fixed pixel grids, 3D structures are mostly empty space. Using standard voxel grids is computationally prohibitive ($O(N^3)$ complexity). Furthermore, while autoregressive models (like GPT) excel at 1D sequences, applying them to sparse 3D data is difficult because the "next" token's position is dynamic and unpredictable, making it hard for the model to learn spatial relationships.
Methodology: The Uni-3DAR Framework
Uni-3DAR treats 3D modeling as a sequence-to-sequence problem through three key innovations:
- Coarse-to-Fine Octree Tokenizer: Instead of a dense grid, the model uses an octree—a tree structure that recursively subdivides 3D space only where data exists. This compresses sparse 3D structures into compact 1D sequences.
- 2-Level Subtree Compression: To further shorten the sequence, the authors group eight subcells (a $2 \times 2 \times 2$ cube) into a single 8-bit token representing 256 possible occupancy states. This reduces the sequence length by roughly 8x.
- Masked Next-Token Prediction (MNTP): This is the "secret sauce" for handling dynamic positions. In standard autoregressive modeling, the model predicts the next token's content and position simultaneously. In MNTP, the model is explicitly given the position of the target token alongside a
[MASK]symbol. It then predicts only the content (e.g., atom type or occupancy). This decouples "where" from "what," significantly stabilizing the training process.
Datasets and Evaluation
The model was evaluated across a massive range of benchmarks to prove its "cross-scale" versatility:
- Microscopic (Molecules & Crystals): QM9 and GEOM-DRUG for small molecules; Carbon-24, MP-20, and MPTS-52 for crystal structures.
- Biological (Proteins): CASF-2016, PDBBind v2020, and MOAD for protein pocket prediction and molecular docking.
- Macroscopic (Objects): ShapeNet (specifically the airplane, chair, and car categories) for 3D shape generation.
- Pretraining: A massive dataset of 19 million molecules was used for structure-level property prediction tasks.
Key Results and Insights
Uni-3DAR demonstrates a significant leap over current diffusion-based state-of-the-art (SOTA) models. It achieves inference speeds up to 21.8x faster than GeoLDM. In crystal structure prediction, it showed a 256% relative improvement in accuracy. On the QM9 dataset, it reached 93.7% molecule stability, surpassing the previous best of ~89%.
Beyond metrics, the work suggests a "GPT Moment" for 3D modeling. By tokenizing 3D space effectively, the same Transformer architecture can handle everything from angstrom-scale atoms to meter-scale furniture. The octree provides a natural "curriculum," where the model learns coarse shapes first and then fills in fine-grained details, mirroring human conceptualization of 3D design.
Conclusion and Future Work
While the framework is unified, current resource constraints meant training separate instances for different benchmarks rather than a single massive foundation model. Future work involves scaling up to a truly heterogeneous mixture of 3D data to investigate "cross-domain" knowledge transfer. The authors also suggest extending the framework to more complex real-world applications like climate forecasting or fluid dynamics.
Summary updated