TL;DR
OmniLatent is a unified framework that jointly models 3D geometry, appearance, and part structure within a single, shared latent space. By leveraging data-rich geometric and visual priors to support data-scarce part learning, it enables high-quality, part-aware 3D asset generation and interactive editing from just a single image.
What Problem Does This Paper Address?
Creating editable 3D assets usually requires breaking them down into functional parts (e.g., the wheels, body, and windows of a car). Traditionally, this "part decomposition" is treated as a post-processing step: you generate a mesh first, then run a separate segmentation algorithm.
This decoupled approach has several flaws. First, the segmenter is at the mercy of the mesh quality; if the geometry is messy, the parts will be too. Second, it's hard to maintain consistency between texture boundaries and part boundaries. Finally, while we have millions of raw 3D shapes, high-quality part annotations are rare. OmniLatent addresses these issues by treating part decomposition not as an isolated labeling task, but as an integral part of the generative process.
Methods
The core of OmniLatent is a unified latent modeling approach. Instead of separate models for shape and parts, everything is projected into the same "OmniLatent" space.
- Unified VAE: The authors developed a Variational Autoencoder that compresses heterogeneous data—Signed Distance Fields (SDF) for geometry, vertex offsets for deformation, RGB values for appearance, and 6-bit binary IDs for parts—into a shared set of latent tokens.
- MM-DiT (Multi-Modal Diffusion Transformer): A generative backbone that takes a single image and predicts these latent tokens. Because the tokens are shared, the model naturally learns that part boundaries should align with geometric edges and texture transitions.
- Progressive Training: To handle the data scarcity of part labels, the model is trained in stages: first on massive amounts of raw geometry, then on textured assets, and finally on a smaller set of part-annotated objects.
- Interactive Extensions: The researchers "tuned" the base model for two specific workflows:
- Point-Prompted Control: Users can click a few points on a 3D surface to define or refine parts.
- Amodal Part Completion: Given partial surface masks, the model can infer the full, "amodal" 3D volume of a part, even the bits hidden inside or behind the object.
Datasets
- Geometry/Deformation: 900K high-quality meshes curated from Objaverse.
- Appearance: 300K assets containing both geometry and texture.
- Part-Level Modeling: A specialized set of 20K part-aware 3D objects, manually annotated with semantic IDs.
- Evaluation: The model was tested on PartObjaverse-Tiny, a benchmark of ~3,000 fine-grained part annotations across 200 shapes.
Key Insights and Ideas
- Geometry as a Prior: The "aha moment" here is that geometry and appearance provide the "structural evidence" for parts. By training them together, the model uses its deep understanding of 3D shapes to make better guesses about where one part ends and another begins.
- Binary over RGB: Most previous works used RGB colors to represent parts. OmniLatent uses 6-bit binary codes. Why? Because during the "denoising" process of diffusion, small color deviations in RGB space can accidentally split one part into many. Bitwise decoding is much more stable and preserves categorical precision.
- Dynamic Token Allocation: For amodal completion, the model doesn't give every part the same amount of "brain power." It allocates more latent tokens to larger parts, allowing it to scale effectively across different object types.
Key Results
OmniLatent significantly outperforms existing "post-hoc" segmentation methods:
- Accuracy: On the PartObjaverse-Tiny benchmark, it achieved 74.18 mIoU (Mean Intersection over Union), a massive jump over the previous state-of-the-art, P³-SAM, which scored 58.10.
- Efficiency: It generates a complete, part-aware asset in about 30 seconds. A traditional pipeline (generating geometry, then texture, then segmenting) takes roughly 80 seconds.
- Interactive Editing: The point-prompt adapter allows for complex operations like "part subdivision" (splitting a coarse region into finer pieces) with just a few clicks, outperforming specialized interactive models like Point-SAM.
Limitations and Future Work
Despite its strengths, OmniLatent has a few "known issues":
- Symmetry: The model can sometimes produce asymmetric part layouts on perfectly symmetric objects (e.g., one wing of a plane segmented differently than the other).
- Mismerging: It occasionally merges two distinct parts if they share very similar textures and geometry.
- View Dependency: When guided by 2D masks, the model can sometimes struggle with "back-side" errors where a 2D split doesn't translate perfectly to the 3D structure.
The authors suggest that future work could incorporate explicit symmetry priors or extend the unified latent space to include even more attributes like physical materials or articulation (how the parts move).
Summary updated