SAGNet: structure-aware generative network for 3D-shape modeling

Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, Hui Huang

Abstract

Generating realistic 3D shapes requires a fundamental understanding of the "logic" behind object composition. A chair, for instance, is a functional assembly of a seat, back, and legs, where the geometry of each part is intrinsically linked to its structural role. We introduce SAGNet (Structure-aware Generative Network), a generative model that jointly learns part geometry and global structure. By employing a two-branch autoencoder with cross-domain attention, SAGNet creates a unified latent space that captures the interdependencies between what a part looks like and where it belongs. Our evaluation demonstrates that SAGNet achieves superior fidelity and structural coherence compared to existing state-of-the-art methods.

Introduction

In 3D computer graphics, generative models traditionally prioritize either local geometric detail or global structural arrangement. Geometry-focused models often produce realistic "blobs" that lack functional logic, while structure-focused models generate well-arranged boxes that lack unique geometric character. The core challenge lies in the fact that geometry and structure are interdependent: a specific joint's shape dictates its placement, and vice versa.

Overview of the SAGNet framework, illustrating joint analysis of geometry and structure for latent space exploration.

SAGNet addresses this by explicitly modeling the relationship between these two domains. By intertwining geometric features with structural context, the model learns a "DNA" for 3D shapes that ensures every generated part is both visually plausible and structurally sound.

Methodology

The architecture of SAGNet is built upon a two-branch autoencoder designed for bidirectional information exchange between geometry and structure.

Detailed architecture of the two-branch autoencoder with geometry and structure attention modules.

Dual-Branch Encoding

  • Geometry Branch: This branch utilizes 3D convolutions to extract high-dimensional features from voxel representations of individual semantic parts.
  • Structure Branch: Parallel to the geometry branch, this component uses fully connected (FC) layers to process structural data, specifically the pairwise relationships and bounding box parameters of the parts.

Information Exchange via Attention

The defining feature of SAGNet is the Geometry and Structure Attention modules. During the encoding process, these modules allow the geometry branch to "attend" to structural information and the structure branch to incorporate geometric context. This ensures that the resulting features are context-aware, reflecting how a part's shape is influenced by its neighbors.

2-Way VAE and Latent Space

These intertwined features are fused into a single 512-dimensional latent code via a Variational Autoencoder (VAE). This unified latent space allows for smooth interpolation between shapes, where transitions are logically consistent in both appearance and arrangement.

Experiments and Results

The researchers evaluated SAGNet using a synthetic Tenon-Mortise Joint dataset—designed to test the model's ability to learn precise geometric fits—and the ShapeNet dataset for real-world object classes like chairs, airplanes, and lamps.

Key Insights

  • Joint Analysis is Essential: Experiments on the tenon-mortise dataset proved that without the attention-based exchange, models fail to learn that the shape of a cavity (geometry) is strictly dependent on the part it must receive (structure).
  • Bidirectional Hallucination: SAGNet can successfully perform "Structure-to-Geometry" mapping, inferring detailed part shapes from simple bounding boxes, and vice versa, with high accuracy.

Quantitative Performance

On the segmented Chair dataset, SAGNet outperformed baselines such as G2L and GRASS. It achieved a Minimum Matching Distance (MMD-CD) of 0.0024, significantly lower than G2L (0.0034) and GRASS (0.0030), indicating higher reconstruction accuracy. Furthermore, higher Coverage (COV) scores demonstrated that SAGNet generates a more diverse set of plausible shapes.

Conclusion

SAGNet represents a significant advancement in 3D shape modeling by treating geometry and structure as a unified whole. While currently limited by fixed part counts and voxel resolution, the framework lays the groundwork for more complex generative tasks. Future research will explore point cloud representations and variable part counts to further enhance the flexibility and detail of the generated models.

Summary updated

Abstract

We present SAGNet, a structure-aware generative model for 3D shapes. Given a set of segmented objects of a certain class, the geometry of their parts and the pairwise relationships between them (the structure) are jointly learned and embedded in a latent space by an autoencoder. The encoder intertwines the geometry and structure features into a single latent code, while the decoder disentangles the features and reconstructs the geometry and structure of the 3D model. Our autoencoder consists of two branches, one for the structure and one for the geometry. The key idea is that during the analysis, the two branches exchange information between them, thereby learning the dependencies between structure and geometry and encoding two augmented features, which are then fused into a single latent code. This explicit intertwining of information enables separately controlling the geometry and the structure of the generated models. We evaluate the performance of our method and conduct an ablation study. We explicitly show that encoding of shapes accounts for both similarities in structure and geometry. A variety of quality results generated by SAGNet are presented.
BibTeX
@article{wu2019sagnet,
  author = {Zhijie Wu and Xiang Wang and Di Lin and Dani Lischinski and Daniel Cohen-Or and Hui Huang},
  title = {{SAGNet: structure-aware generative network for 3D-shape modeling}},
  journal = {ACM Trans. Graphics},
  year = {2019},
  url = {https://openreview.net/forum?id=ANnMCfdmF_F},
}
Figure