OmniLatent: Unified Latent for 3D Part Modeling

Yunhan Yang, Yunfei Zhao, Zeqiang Lai, Jiachen Xu, Minxiu Wang, Xintong Han, Xiangyu Yue, Xihui Liu, Chunchao GuoarXiv 2026 · arXiv

TL;DR

OmniLatent is a unified framework that jointly models 3D geometry, appearance, and part structure within a single, shared latent space. By leveraging data-rich geometric and visual priors to support data-scarce part learning, it enables high-quality, part-aware 3D asset generation and interactive editing from just a single image.

What Problem Does This Paper Address?

Creating editable 3D assets usually requires breaking them down into functional parts (e.g., the wheels, body, and windows of a car). Traditionally, this "part decomposition" is treated as a post-processing step: you generate a mesh first, then run a separate segmentation algorithm.

This decoupled approach has several flaws. First, the segmenter is at the mercy of the mesh quality; if the geometry is messy, the parts will be too. Second, it's hard to maintain consistency between texture boundaries and part boundaries. Finally, while we have millions of raw 3D shapes, high-quality part annotations are rare. OmniLatent addresses these issues by treating part decomposition not as an isolated labeling task, but as an integral part of the generative process.

Methods

The core of OmniLatent is a unified latent modeling approach. Instead of separate models for shape and parts, everything is projected into the same "OmniLatent" space.

  1. Unified VAE: The authors developed a Variational Autoencoder that compresses heterogeneous data—Signed Distance Fields (SDF) for geometry, vertex offsets for deformation, RGB values for appearance, and 6-bit binary IDs for parts—into a shared set of latent tokens.
  2. MM-DiT (Multi-Modal Diffusion Transformer): A generative backbone that takes a single image and predicts these latent tokens. Because the tokens are shared, the model naturally learns that part boundaries should align with geometric edges and texture transitions.
  3. Progressive Training: To handle the data scarcity of part labels, the model is trained in stages: first on massive amounts of raw geometry, then on textured assets, and finally on a smaller set of part-annotated objects.
  4. Interactive Extensions: The researchers "tuned" the base model for two specific workflows:
  5. Point-Prompted Control: Users can click a few points on a 3D surface to define or refine parts.
  6. Amodal Part Completion: Given partial surface masks, the model can infer the full, "amodal" 3D volume of a part, even the bits hidden inside or behind the object.
The OmniLatent architecture uses a VAE to encode multiple attributes into a shared space, which is then sampled by a Multi-Modal Diffusion Transformer.

Datasets

  • Geometry/Deformation: 900K high-quality meshes curated from Objaverse.
  • Appearance: 300K assets containing both geometry and texture.
  • Part-Level Modeling: A specialized set of 20K part-aware 3D objects, manually annotated with semantic IDs.
  • Evaluation: The model was tested on PartObjaverse-Tiny, a benchmark of ~3,000 fine-grained part annotations across 200 shapes.

Key Insights and Ideas

  • Geometry as a Prior: The "aha moment" here is that geometry and appearance provide the "structural evidence" for parts. By training them together, the model uses its deep understanding of 3D shapes to make better guesses about where one part ends and another begins.
  • Binary over RGB: Most previous works used RGB colors to represent parts. OmniLatent uses 6-bit binary codes. Why? Because during the "denoising" process of diffusion, small color deviations in RGB space can accidentally split one part into many. Bitwise decoding is much more stable and preserves categorical precision.
  • Dynamic Token Allocation: For amodal completion, the model doesn't give every part the same amount of "brain power." It allocates more latent tokens to larger parts, allowing it to scale effectively across different object types.
Fig. 1. OmniLatent generates geometry, deformation, appearance, and coherent surface part IDs from a single image in a unified latent space.

Key Results

OmniLatent significantly outperforms existing "post-hoc" segmentation methods:

  • Accuracy: On the PartObjaverse-Tiny benchmark, it achieved 74.18 mIoU (Mean Intersection over Union), a massive jump over the previous state-of-the-art, P³-SAM, which scored 58.10.
  • Efficiency: It generates a complete, part-aware asset in about 30 seconds. A traditional pipeline (generating geometry, then texture, then segmenting) takes roughly 80 seconds.
  • Interactive Editing: The point-prompt adapter allows for complex operations like "part subdivision" (splitting a coarse region into finer pieces) with just a few clicks, outperforming specialized interactive models like Point-SAM.

Limitations and Future Work

Despite its strengths, OmniLatent has a few "known issues":

  • Symmetry: The model can sometimes produce asymmetric part layouts on perfectly symmetric objects (e.g., one wing of a plane segmented differently than the other).
  • Mismerging: It occasionally merges two distinct parts if they share very similar textures and geometry.
  • View Dependency: When guided by 2D masks, the model can sometimes struggle with "back-side" errors where a 2D split doesn't translate perfectly to the 3D structure.

The authors suggest that future work could incorporate explicit symmetry priors or extend the unified latent space to include even more attributes like physical materials or articulation (how the parts move).

Summary updated

Abstract

3D part decomposition is a central capability for editable asset creation, yet existing workflows typically segment a completed mesh as a post-processing step, making the result sensitive to geometric artifacts, texture ambiguity, and limited part-level annotations. We present OmniLatent, a unified latent modeling framework for 3D part decomposition. Instead of predicting part labels in isolation, OmniLatent learns a shared latent space that jointly encodes geometry, deformation, appearance, and binary part IDs, allowing data-rich geometric and visual priors to support data-scarce part learning. Given a single input image, the base model generates geometry, deformation, texture, and surface part IDs in one pass, producing a part-aware 3D asset without a separate segmentation stage. We further tune the pretrained model for two extensions: a point-prompt adapter that maps arbitrary 3D point prompts with binary part IDs into latent tokens for full segmentation, refinement, and part subdivision, and an amodal part completion model that completes partial surface masks into a coherent full-object decomposition through dynamic per-part token allocation, cross-attention to the given geometry latent, and alternating part-local and global attention. Extensive experiments show that OmniLatent produces semantically meaningful part segmentations with precise boundaries, supports interactive part editing, and completes partial part observations while preserving global object consistency.
BibTeX
@misc{yang2026omnilatent,
  author = {Yunhan Yang and Yunfei Zhao and Zeqiang Lai and Jiachen Xu and Minxiu Wang and Xintong Han and Xiangyu Yue and Xihui Liu and Chunchao Guo},
  title = {{OmniLatent: Unified Latent for 3D Part Modeling}},
  year = {2026},
  eprint = {2611.09055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09055},
}
Figure