Introduction
Generating high-quality 3D shapes remains a significant challenge in computer vision and graphics. While 2D image generation has reached high levels of maturity, 3D synthesis often struggles with "blobby" geometry, broken connectivity, and a lack of fine-grained structural control. Traditional generative models—whether based on voxels, point clouds, or implicit fields—frequently fail to capture the underlying topology of complex objects.
GEM3D addresses these limitations by introducing a generative framework that utilizes "medial abstractions" as an intermediate representation. By first synthesizing a skeletal structure based on the Medial Axis Transform (MAT) and then "fleshing it out" into a full surface, the model achieves state-of-the-art results in generating topologically complex shapes with intuitive, skeleton-based control.
The Challenge of 3D Topology
Most current 3D generative models operate as black boxes. While they can produce plausible volumes, they often fail to resolve intricate details, such as the individual slats of a chair or the thin wires of a lamp. These models frequently merge distinct parts into a single solid mass or leave them disconnected, primarily because they lack explicit spatial grounding. Humans naturally understand shapes through their structural "backbone," but neural networks often lack this structural prior. GEM3D makes the skeleton a first-class citizen in the generative process to solve these topological inconsistencies.
Methodology
GEM3D employs a two-stage diffusion process to bridge the gap between abstract structure and detailed surface geometry.
1. Skeleton Generation
In the first stage, the model uses a denoising diffusion probabilistic model to generate a point-based skeletal representation. This representation is based on the Medial Axis Transform (MAT), which reduces a 3D volume to its central lines and sheets. This stage effectively "decides" the connectivity and essential structure of the shape before any surface details are considered.
2. Latent Synthesis and Neural Enveloping
A second diffusion stage generates spatially grounded latent codes for every point on the generated skeleton. These codes store information regarding local thickness and surface characteristics. Finally, a specialized neural implicit decoder—utilizing a "closest envelope" approach—shrink-wraps a surface around the skeleton. Unlike older methods that only consider the nearest skeletal point, this approach considers the envelopes of all nearby points, preventing "ball-like" artifacts and enabling the synthesis of smooth, complex surfaces.
Experimental Evaluation
Datasets
The researchers evaluated GEM3D using two primary benchmarks:
- ShapeNet-v2: The model was tested across the 10 largest categories (e.g., chairs, planes, lamps) and validated across all 55 categories.
- Thingi10K: This dataset, comprised of 3D-printing models with high genus (complex topology), served as a stress test for out-of-distribution performance.
Key Results
GEM3D consistently outperformed existing models such as 3DILG and 3DS2VS:
- Fidelity and Diversity: On ShapeNet, GEM3D achieved a Recall of 89.2%, significantly higher than the previous best of 80.3%, indicating a superior ability to generate a wide variety of valid shapes.
- Topological Robustness: In the Thingi10K stress test, GEM3D maintained holes and thin connections more effectively than competitors. For high-complexity shapes (genus ≥ 20), its Chamfer Distance was 2.32, compared to 2.91 for 3DS2VS.
- User-Driven Synthesis: The model demonstrates the ability to take hand-drawn skeletons and generate plausible, manifold 3D meshes, offering a level of interpretability and control not found in category-only models.
Discussion and Future Work
GEM3D represents a significant advancement in 3D synthesis by grounding generation in medial abstractions. However, the model is not without limitations. Noisy skeletons generated in the first stage can lead to bumpy surface artifacts, and the current point-based skeleton limits the connectivity compared to what might be achieved with a graph-based structure.
Future research may explore moving toward structured graph or mesh-based skeletons to further improve connectivity. Additionally, the potential for interactive skeleton editing—allowing users to manipulate "bones" in real-time to refine generated shapes—presents an exciting frontier for human-AI collaborative design.
Summary updated