TL;DR
RDT-3D introduces Resolution-Decoupled Training, a novel strategy that fixes a fundamental mismatch between how 3D generation models are trained and how they are used at inference. By anchoring spatial guidance at a coarse resolution while supervising geometry at an ultra-high resolution, the framework enables the generation of high-fidelity 3D assets with significantly sharper edges and finer details than current state-of-the-art methods.
What Problem Does This Paper Address?
Modern 3D generation often follows a "coarse-to-fine" approach: a first stage generates a rough global shape, and a second stage refines the local surface details. Many top-tier models use Rotary Position Embeddings (RoPE) to tell the refinement network exactly where in space it is working.
The authors identified a critical "resolution discrepancy" in this process:
- During Training: Models are typically trained symmetrically. If the target output is high-resolution (e.g., 1536³), the positional guidance (RoPE) is also provided at that same high resolution.
- During Inference: The model receives positional guidance from a coarse stage-1 mesh (e.g., 512³), but it is still expected to produce a high-resolution output.
This mismatch creates a severe distribution shift. Because the model only ever saw "perfect" high-res coordinates during training, it struggles to handle the "imprecise" coarse coordinates it gets in the real world, leading to oversmoothed surfaces, rounded edges, and lost thin structures.
Methods
The core innovation is Resolution-Decoupled Training (RDT). Instead of aligning guidance and supervision resolutions, RDT explicitly decouples them:
- Coarse Localization Branch: During training, the RoPE coordinate guidance is anchored at a coarse resolution ($R_g = 512$). This forces the network to handle the same level of spatial "uncertainty" it will encounter during actual use.
- Fine Supervision Branch: Simultaneously, the geometric target (the ground truth) is kept at an ultra-high resolution ($R_s = 1536$).
By training this way, the model learns a robust mapping: it becomes an expert at extrapolating high-frequency details from coarse spatial cues. The architecture utilizes a RoPE-guided Diffusion Transformer (DiT) and a rectified-flow objective for stable optimization.
Datasets
To support this high-resolution training, the authors introduced Dual3D-300K, a massive dataset of 300,000 watertight 3D point cloud pairs.
- Alignment: Each object is explicitly aligned across 512 and 1536 resolutions.
- Curation: Data is sourced from high-quality repositories like ABO, 3D-FUTURE, and Objaverse-XL.
- Sharpness-Aware Sampling: To ensure the model learns to respect fine features, the dataset uses a mix of uniform surface sampling and curvature-concentrated sampling (focusing more points on edges and thin parts).
Key Insights and Ideas
- Structured Noise vs. Random Noise: The authors found that simply adding random Gaussian noise (jitter) to coordinates during training doesn't solve the problem. The resolution gap is a structured shift, and only explicit resolution decoupling (RDT) effectively bridges it.
- Extrapolation is Key: The refinement network shouldn't just be a "copy-paste" mechanism for coordinates; it needs to be an inference engine that predicts what a sharp edge should look like based on a blurry spatial hint.
- Stability through GT: While the model eventually uses stage-1 outputs at inference, the authors train using low-res versions of the ground truth for guidance. This maintains global structural alignment and prevents the training process from becoming unstable due to the "hallucinations" of upstream models.
Key Results
RDT-3D was tested against heavyweights like Hunyuan3D-2.1, TRELLIS, and UltraShape.
- Quantitative: RDT-3D achieved a Chamfer Distance (CD) of 0.0164 and an F-Score of 0.6886, consistently outperforming all baselines.
- Qualitative: The visual difference is striking. RDT-3D produces meshes with crisp, sharp boundaries and intricate topologies (like the scales on a dragon or the fine carvings on a jade ornament) where other models produce "melted" or oversmoothed versions.
Ablation studies confirmed that the RDT strategy is the primary driver of these improvements, as shown in the comparison between the baseline and the RDT-trained version.
Limitations and Future Work
While the paper demonstrates a significant leap in fidelity, a few areas remain for exploration:
- Inference Dependency: The quality of the final high-res mesh is still somewhat tethered to the global topology established by the stage-1 coarse model. If the coarse model misses a major limb or part, the refiner may not be able to "invent" it.
- Computational Cost: Generating at 1536 resolution is computationally intensive. While the Dual3D-300K dataset lowers the barrier for others to enter this space, the inference time for such high-fidelity assets is naturally higher than for lower-res models.
Summary updated