Abstract
Generating realistic 3D shapes requires a fundamental understanding of the "logic" behind object composition. A chair, for instance, is a functional assembly of a seat, back, and legs, where the geometry of each part is intrinsically linked to its structural role. We introduce SAGNet (Structure-aware Generative Network), a generative model that jointly learns part geometry and global structure. By employing a two-branch autoencoder with cross-domain attention, SAGNet creates a unified latent space that captures the interdependencies between what a part looks like and where it belongs. Our evaluation demonstrates that SAGNet achieves superior fidelity and structural coherence compared to existing state-of-the-art methods.
Introduction
In 3D computer graphics, generative models traditionally prioritize either local geometric detail or global structural arrangement. Geometry-focused models often produce realistic "blobs" that lack functional logic, while structure-focused models generate well-arranged boxes that lack unique geometric character. The core challenge lies in the fact that geometry and structure are interdependent: a specific joint's shape dictates its placement, and vice versa.
SAGNet addresses this by explicitly modeling the relationship between these two domains. By intertwining geometric features with structural context, the model learns a "DNA" for 3D shapes that ensures every generated part is both visually plausible and structurally sound.
Methodology
The architecture of SAGNet is built upon a two-branch autoencoder designed for bidirectional information exchange between geometry and structure.
Dual-Branch Encoding
- Geometry Branch: This branch utilizes 3D convolutions to extract high-dimensional features from voxel representations of individual semantic parts.
- Structure Branch: Parallel to the geometry branch, this component uses fully connected (FC) layers to process structural data, specifically the pairwise relationships and bounding box parameters of the parts.
Information Exchange via Attention
The defining feature of SAGNet is the Geometry and Structure Attention modules. During the encoding process, these modules allow the geometry branch to "attend" to structural information and the structure branch to incorporate geometric context. This ensures that the resulting features are context-aware, reflecting how a part's shape is influenced by its neighbors.
2-Way VAE and Latent Space
These intertwined features are fused into a single 512-dimensional latent code via a Variational Autoencoder (VAE). This unified latent space allows for smooth interpolation between shapes, where transitions are logically consistent in both appearance and arrangement.
Experiments and Results
The researchers evaluated SAGNet using a synthetic Tenon-Mortise Joint dataset—designed to test the model's ability to learn precise geometric fits—and the ShapeNet dataset for real-world object classes like chairs, airplanes, and lamps.
Key Insights
- Joint Analysis is Essential: Experiments on the tenon-mortise dataset proved that without the attention-based exchange, models fail to learn that the shape of a cavity (geometry) is strictly dependent on the part it must receive (structure).
- Bidirectional Hallucination: SAGNet can successfully perform "Structure-to-Geometry" mapping, inferring detailed part shapes from simple bounding boxes, and vice versa, with high accuracy.
Quantitative Performance
On the segmented Chair dataset, SAGNet outperformed baselines such as G2L and GRASS. It achieved a Minimum Matching Distance (MMD-CD) of 0.0024, significantly lower than G2L (0.0034) and GRASS (0.0030), indicating higher reconstruction accuracy. Furthermore, higher Coverage (COV) scores demonstrated that SAGNet generates a more diverse set of plausible shapes.
Conclusion
SAGNet represents a significant advancement in 3D shape modeling by treating geometry and structure as a unified whole. While currently limited by fixed part counts and voxel resolution, the framework lays the groundwork for more complex generative tasks. Future research will explore point cloud representations and variable part counts to further enhance the flexibility and detail of the generated models.
Summary updated