JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation

Xiangjun Tang, Jianqi Chen, Peter WonkaarXiv 2026 · arXiv

TL;DR

JoVox is a novel 3D generation framework that produces high-fidelity voxels with unprecedented efficiency. By using joint hierarchical flow matching, it allows finer details to inform coarse structures and predicts two hierarchy levels at once, achieving 8x higher training throughput and 1/15th the FLOPs of the current state-of-the-art, TRELLIS.2, while actually improving shape quality.

What Problem Does This Paper Address?

Generating 3D voxels—essentially 3D pixels—is a balancing act between fidelity and computational cost.

Existing methods generally fall into two camps. Dense/Latent methods (like TRELLIS.2) achieve high fidelity but are computationally "heavy" because they process every cell in a grid, including empty space. Hierarchical methods are "lightweight" because they only refine occupied areas, but they typically work in a strict top-down fashion: coarse levels are decided first with very little information and are never revisited. If a coarse level is slightly off, the error propagates down, leading to shapes that look "plausible but wrong."

JoVox aims to bridge this gap, providing the lightweight efficiency of hierarchical models with the high fidelity of dense ones.

Methods

The core innovation of JoVox is Joint Hierarchical Modeling. Instead of learning each level of the voxel hierarchy separately, JoVox learns the joint distribution of all levels simultaneously using a discrete flow matching framework.

Key technical components include:

  • Joint Distribution Learning: By modeling all levels together, the system allows finer cells to participate in deciding the state of coarser cells. During the iterative generation process, coarse cells that are still "masked" can be predicted based on finer details already unmasked nearby.
  • Two-Level Prediction: Traditional hierarchical models predict one level finer at a time (e.g., an $8^3$ grid from a $4^3$ grid). JoVox predicts two levels finer at once (e.g., jumping straight to a $16^3$ grid). This halves the number of levels the network must process and, surprisingly, yields higher fidelity.
  • Smart Unmasking Strategy: Masked diffusion models often suffer from inconsistencies when multiple tokens are unmasked at once. JoVox introduces a strategy that "remasks" the oldest decisions. Since these were made when the model had the least information, revisiting them once finer details are available allows the model to correct early mistakes.
  • Sparse Transformer Architecture: The model uses a transformer that only processes tokens for occupied cells, drastically reducing memory and compute requirements compared to dense transformers.
The JoVox generation process: finer resolutions start before coarser ones are complete, allowing for a more cohesive final shape.

Datasets

The authors evaluated JoVox on both image-conditioned and unconditional generation tasks:

  • Training: A subset of TRELLIS-500K containing approximately 175,000 3D assets.
  • Image-Conditioned Evaluation: Tested on 1,000 unseen objects from the Toys4k dataset.
  • Unconditional Evaluation: Tested on four categories (chair, airplane, table, rifle) from the ShapeNet dataset.

Key Insights and Ideas

  • Finer-to-Coarse Feedback: The "aha moment" here is that hierarchy shouldn't be a one-way street. By allowing the model to look at finer-scale predictions to refine coarse-scale ones, JoVox avoids the "plausible but wrong" trap of traditional octree-based generators.
  • The Efficiency of Sparsity: Because JoVox only places tokens where the shape actually exists, it avoids wasting billions of operations on empty space. This is why it can run on significantly less hardware than its competitors.
  • Granularity Jumps Matter: The move to two-level prediction isn't just a speed hack; it provides more local context to the model at each step, which the authors found actually improved the geometric accuracy of the resulting voxels.
Figure 2. (a) Two-level prediction. For each cell u at level l, the network predicts the occupancy codes of the eight children of u at level l + 1. The occupancy code of the j-th child consists of the values aj,1, . . . , aj,8 of its eight children at level l + 2, where a value is 1 for an occupied cell, 0 for an empty one and ∅for a cell whose occupancy is unknown. When this occupancy code is masked, the value aj of the j-th child and aj,1, . . . , aj,8 are all ∅. (b) The network. Each cell u at level 0 and each occupied cell u at even levels has one token. The token h embeds the values aj and aj,b, the level l and the position r. A transformer processes the tokens of all levels together, conditioned on c and on the fractions t of unmasked occupancy codes, and outputs for u eight distributions over the 256 values of an occupancy code.

Key Results

JoVox demonstrates a massive leap in efficiency without sacrificing quality:

  • Efficiency: It achieves 8x higher training throughput and uses only 1/15th of the FLOPs in a forward pass compared to TRELLIS.2.
  • Hardware Accessibility: While TRELLIS.2 required 32 H100 GPUs for training, JoVox was trained on just 4 GPUs.
  • Geometric Accuracy: On the Toys4k benchmark, JoVox achieved an 11% lower Chamfer distance (a measure of how closely the generated shape matches the ground truth) and a higher Voxel IoU (0.416 vs 0.402) than TRELLIS.2.
  • Qualitative Superiority: As seen in Figure 3, JoVox produces sharper details and more consistent structures, especially in complex areas like the oars of a boat or the legs of a chair.
Qualitative comparison showing JoVox producing more accurate geometry than Direct3D-S2, DVD, and TRELLIS variants.

Limitations and Future Work

  • Training Data Scale: The model was trained on less than a quarter of the data used for TRELLIS.2 due to resource constraints. The authors expect even better performance if scaled to the full 800K+ asset dataset.
  • Independent Sampling Issues: Like other masked discrete models, JoVox samples tokens independently in each step, which can occasionally lead to minor local inconsistencies.
  • Scaling to Scenes: Currently focused on single objects, the authors plan to extend the framework to generate large, complex scenes containing multiple objects, where the lightweight nature of JoVox would be even more advantageous.

Summary updated

Abstract

Generating high-fidelity voxels while keeping their lightweight nature remains an open problem in computer vision. Current generators are either high-fidelity but computationally heavy, or lightweight but of lower fidelity. Hierarchical methods keep the lightweight nature of voxels. However, their coarse levels, which decide where all finer cells can appear, are generated with the least information about the shape and are never revisited, so they are often plausible but wrong. Our key insight is to let finer cells take part in deciding coarse cells. To this end, we propose JoVox, a discrete flow matching framework that learns the joint distribution of all levels of the voxel hierarchy, instead of a separate distribution for each level, so that coarse cells can be predicted with finer cells already generated nearby. Within this framework, we further introduce a two-level prediction, which outputs the cells two levels finer at once instead of one level finer, halving the number of levels that the network processes and yielding voxels of higher fidelity. We also introduce an unmasking strategy that revisits early decisions once finer cells are generated and corrects inconsistencies in masked diffusion models. Our framework outperforms the baselines on both unconditional and image-conditioned voxel generation. Compared with the current state-of-the-art TRELLIS.2, we achieve 8x higher training throughput, 1/15 of the FLOPs in a forward pass, train on 4 vs. 32 GPUs, and obtain higher quality with an 11% lower Chamfer distance.
BibTeX
@misc{tang2026jovox,
  author = {Xiangjun Tang and Jianqi Chen and Peter Wonka},
  title = {{JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation}},
  year = {2026},
  eprint = {2611.10690},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10690},
}
Figure