TL;DR
JoVox is a novel 3D generation framework that produces high-fidelity voxels with unprecedented efficiency. By using joint hierarchical flow matching, it allows finer details to inform coarse structures and predicts two hierarchy levels at once, achieving 8x higher training throughput and 1/15th the FLOPs of the current state-of-the-art, TRELLIS.2, while actually improving shape quality.
What Problem Does This Paper Address?
Generating 3D voxels—essentially 3D pixels—is a balancing act between fidelity and computational cost.
Existing methods generally fall into two camps. Dense/Latent methods (like TRELLIS.2) achieve high fidelity but are computationally "heavy" because they process every cell in a grid, including empty space. Hierarchical methods are "lightweight" because they only refine occupied areas, but they typically work in a strict top-down fashion: coarse levels are decided first with very little information and are never revisited. If a coarse level is slightly off, the error propagates down, leading to shapes that look "plausible but wrong."
JoVox aims to bridge this gap, providing the lightweight efficiency of hierarchical models with the high fidelity of dense ones.
Methods
The core innovation of JoVox is Joint Hierarchical Modeling. Instead of learning each level of the voxel hierarchy separately, JoVox learns the joint distribution of all levels simultaneously using a discrete flow matching framework.
Key technical components include:
- Joint Distribution Learning: By modeling all levels together, the system allows finer cells to participate in deciding the state of coarser cells. During the iterative generation process, coarse cells that are still "masked" can be predicted based on finer details already unmasked nearby.
- Two-Level Prediction: Traditional hierarchical models predict one level finer at a time (e.g., an $8^3$ grid from a $4^3$ grid). JoVox predicts two levels finer at once (e.g., jumping straight to a $16^3$ grid). This halves the number of levels the network must process and, surprisingly, yields higher fidelity.
- Smart Unmasking Strategy: Masked diffusion models often suffer from inconsistencies when multiple tokens are unmasked at once. JoVox introduces a strategy that "remasks" the oldest decisions. Since these were made when the model had the least information, revisiting them once finer details are available allows the model to correct early mistakes.
- Sparse Transformer Architecture: The model uses a transformer that only processes tokens for occupied cells, drastically reducing memory and compute requirements compared to dense transformers.
Datasets
The authors evaluated JoVox on both image-conditioned and unconditional generation tasks:
- Training: A subset of TRELLIS-500K containing approximately 175,000 3D assets.
- Image-Conditioned Evaluation: Tested on 1,000 unseen objects from the Toys4k dataset.
- Unconditional Evaluation: Tested on four categories (chair, airplane, table, rifle) from the ShapeNet dataset.
Key Insights and Ideas
- Finer-to-Coarse Feedback: The "aha moment" here is that hierarchy shouldn't be a one-way street. By allowing the model to look at finer-scale predictions to refine coarse-scale ones, JoVox avoids the "plausible but wrong" trap of traditional octree-based generators.
- The Efficiency of Sparsity: Because JoVox only places tokens where the shape actually exists, it avoids wasting billions of operations on empty space. This is why it can run on significantly less hardware than its competitors.
- Granularity Jumps Matter: The move to two-level prediction isn't just a speed hack; it provides more local context to the model at each step, which the authors found actually improved the geometric accuracy of the resulting voxels.
Key Results
JoVox demonstrates a massive leap in efficiency without sacrificing quality:
- Efficiency: It achieves 8x higher training throughput and uses only 1/15th of the FLOPs in a forward pass compared to TRELLIS.2.
- Hardware Accessibility: While TRELLIS.2 required 32 H100 GPUs for training, JoVox was trained on just 4 GPUs.
- Geometric Accuracy: On the Toys4k benchmark, JoVox achieved an 11% lower Chamfer distance (a measure of how closely the generated shape matches the ground truth) and a higher Voxel IoU (0.416 vs 0.402) than TRELLIS.2.
- Qualitative Superiority: As seen in Figure 3, JoVox produces sharper details and more consistent structures, especially in complex areas like the oars of a boat or the legs of a chair.
Limitations and Future Work
- Training Data Scale: The model was trained on less than a quarter of the data used for TRELLIS.2 due to resource constraints. The authors expect even better performance if scaled to the full 800K+ asset dataset.
- Independent Sampling Issues: Like other masked discrete models, JoVox samples tokens independently in each step, which can occasionally lead to minor local inconsistencies.
- Scaling to Scenes: Currently focused on single objects, the authors plan to extend the framework to generate large, complex scenes containing multiple objects, where the lightweight nature of JoVox would be even more advantageous.
Summary updated