ShapeFlow: Learnable Deformations Among 3D Shapes

Chiyu Max Jiang, Jingwei Huang, Andrea Tagliasacchi, Leonidas J. Guibas

Abstract

We present ShapeFlow, a flow-based model for learning a deformation space for entire classes of 3D shapes with large intra-class variations. ShapeFlow allows learning a multi-template deformation space that is agnostic to shape topology, yet preserves fine geometric details. Different from a generative space where a latent vector is directly decoded into a shape, a deformation space decodes a vector into a continuous flow that can advect a source shape towards a target. Such a space naturally allows the disentanglement of geometric style (coming from the source) and structural pose (conforming to the target). We parametrize the deformation between geometries as a learned continuous flow field via a neural network and show that such deformations can be guaranteed to have desirable properties, such as be bijectivity, freedom from self-intersections, or volume preservation. We illustrate the effectiveness of this learned deformation space for various downstream applications, including shape generation via deformation, geometric style transfer, unsupervised learning of a consistent parameterization for entire classes of shapes, and shape interpolation.

Introduction

Traditional 3D representations in computer vision often face a trade-off between geometric detail and topological flexibility. Template-based models are excellent at preserving fine details but require consistent mesh connectivity across all instances. Conversely, generative models using voxels or implicit functions can represent arbitrary topologies but often result in "blobby" reconstructions that lose sharp CAD-like features. ShapeFlow addresses this gap by learning a calligraphic deformation space rather than a generative one, focusing on the continuous transformations between shapes to maintain high-fidelity geometric details.

Methodology

The core of ShapeFlow is treating deformation as an advection process. A 3D shape is morphed from a source to a target configuration by following a learned continuous vector field, effectively "blowing" the geometry into its new form.

Overview of the ShapeFlow framework, illustrating input observations, embedding into a learned deformation space, and various applications.

Neural Flow Fields and Hub-and-Spoke Architecture

Deformations are parameterized as a continuous flow field $f_\theta(x, t)$ learned by a neural network. To handle the complexity of diverse shape categories (e.g., "all chairs"), the model utilizes a "hub-and-spoke" architecture. Every shape is mapped to a central "hub" or canonical latent code, avoiding the computational impossibility of learning pairwise transformations between every shape in a dataset.

Physical Constraints and Embedding

The framework incorporates physical constraints directly into the architecture to ensure realistic results:

  • Invertibility: Ensures that deforming from Shape A to B and back to A returns to the original state.
  • Volume Conservation: For organic models like humans, the flow is parameterized as the curl of a neural network's output, ensuring the volume remains constant during deformation.
  • Auto-decoder Approach: Instead of a separate encoder, latent codes are optimized directly for each shape during training, allowing for precise embedding in the deformation space.

Key Insights and Applications

ShapeFlow's deformation-centric approach enables several unique capabilities that traditional generative models struggle with:

  • Disentangling Style and Structure: "Style" is inherited from the source mesh (topology, sharp edges), while "structure" is dictated by the target (overall pose or proportions).
  • Identity Preservation: Unlike traditional generative decoders that often blur inputs, deforming a shape to the hub and back preserves its original identity almost perfectly.
  • Unsupervised Correspondence: By routing shapes through a common hub, the model naturally learns semantic correspondences across a class without manual labels.
Applications of ShapeFlow, including shape correspondence, reconstruction from sparse/partial data, and geometric style transfer.

Evaluation and Results

The authors evaluated ShapeFlow on ShapeNet (rigid objects) and SMPL (human bodies), demonstrating its versatility across different domains.

  • Surface Reconstruction: ShapeFlow matches or exceeds state-of-the-art methods like OccFlow in reconstructing shapes from sparse point clouds, specifically excelling at preserving sharp corners and flat surfaces.
  • Semantic Matching: On the ShapeNet chair dataset, ShapeFlow achieved a Semantic Matching Score (SMS) of 0.816, significantly higher than standard nearest-neighbor matching (0.779).
  • Physical Plausibility: The volume-preservation constraint effectively prevents the "shrinking" effect common in linear interpolation, maintaining a near-constant volume ratio of 1.0.

Conclusion

ShapeFlow provides a robust framework for 3D modeling that respects both topological diversity and geometric precision. While it faces challenges regarding computational cost and occasional semantic misalignment in unsupervised settings, it represents a significant step toward high-fidelity 3D shape manipulation and reconstruction.

Summary updated

Abstract

We present ShapeFlow, a flow-based model for learning a deformation space for entire classes of 3D shapes with large intra-class variations. ShapeFlow allows learning a multi-template deformation space that is agnostic to shape topology, yet preserves fine geometric details. Different from a generative space where a latent vector is directly decoded into a shape, a deformation space decodes a vector into a continuous flow that can advect a source shape towards a target. Such a space naturally allows the disentanglement of geometric style (coming from the source) and structural pose (conforming to the target). We parametrize the deformation between geometries as a learned continuous flow field via a neural network and show that such deformations can be guaranteed to have desirable properties, such as be bijectivity, freedom from self-intersections, or volume preservation. We illustrate the effectiveness of this learned deformation space for various downstream applications, including shape generation via deformation, geometric style transfer, unsupervised learning of a consistent parameterization for entire classes of shapes, and shape interpolation.
BibTeX
@inproceedings{jiang2020shapeflow,
  author = {Chiyu Max Jiang and Jingwei Huang and Andrea Tagliasacchi and Leonidas J. Guibas},
  title = {{ShapeFlow: Learnable Deformations Among 3D Shapes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Jpl-18iaKg},
}
Figure