OperatorNet: Recovering 3D Shapes From Difference Operators

Ruqi Huang, Marie-Julie Rakotosaona, Panos Achlioptas, Leonidas J. Guibas, Maks Ovsjanikov

TL;DR

This paper introduces OperatorNet, a novel deep learning framework that reconstructs 3D shapes from compact matrix representations called "shape difference operators." This approach significantly outperforms existing geometric methods and enables new applications like shape interpolation and analogy by leveraging the inherent algebraic structure of these operators.

What Problem Does This Paper Address?

Reconstructing and manipulating 3D shapes is a fundamental challenge in computer graphics and vision. Unlike images, 3D shapes lack a regular grid structure, making them difficult to process with standard deep learning techniques like CNNs. Existing methods often struggle with variations in shape topology, pose, and scale, and can produce unrealistic results when interpolating between shapes. Furthermore, representing the intrinsic geometry of a shape while also capturing extrinsic, pose-dependent information has been a persistent challenge.

Methods

OperatorNet tackles these issues by representing shapes as a set of "shape difference operators." These operators, encoded as small matrices, capture how a shape deviates from a base shape. The core of the method involves:

  • Shape Difference Operators: The paper builds upon existing "intrinsic" shape difference operators and introduces a novel "extrinsic" operator. The intrinsic operators capture pose-invariant geometric differences, while the extrinsic operator accounts for pose-dependent information.
  • OperatorNet Architecture: A convolutional neural network (CNN) is designed to take these shape difference matrices as input. A shallow convolutional encoder extracts features, followed by a dense decoder that outputs the 3D shape's coordinate functions.
  • Functional Maps: The framework relies on functional maps, which provide a way to relate functions defined on different shapes, even if they have different discretizations (e.g., different numbers of vertices). This allows for robust comparisons and reconstructions.
  • Algebraic Operations for Synthesis: The paper highlights the multiplicative algebra of shape difference operators. This property allows for synthesizing new shapes through operations like interpolation and analogy, going beyond simple linear interpolation in latent spaces.

Datasets

The paper uses two primary datasets for training and evaluation:

  • Human Shapes:
    • DFAUST dataset: Contains 9440 scanned human characters with various motions.
  • SURREAL dataset: Generates human shapes with diverse body types.
  • Animal Shapes:
    • SMAL dataset: Used to generate 1800 animals (lions, dogs, horses) using a parametric model.

The meshes are simplified to 1000 vertices for humans and 1769 for animals. For experiments, shape differences are constructed using a truncated eigenbasis of dimension 60 on the base shape and the full basis on the target shape.

Key Insights and Ideas

  • Shape Differences as a Powerful Representation: The paper demonstrates that shape difference operators, when encoded as matrices, are a highly effective representation for deep learning, capturing both intrinsic and extrinsic shape information.
  • Algebraic Structure Enables Realistic Synthesis: The multiplicative algebra of shape difference operators is key to generating smooth and realistic shape interpolations and analogies, outperforming methods that rely on linear interpolation in latent spaces.
  • Extrinsic Operator for Pose Robustness: The introduction of a novel extrinsic shape difference operator significantly improves the ability to reconstruct shapes with varying poses.
  • Learning from Collections: By leveraging a collection of shapes, OperatorNet learns to constrain reconstructions to the space of realistic shapes, a significant advantage over purely geometric methods.
  • Handling Different Discretizations: The framework's reliance on functional maps allows it to reconstruct shapes even when the input and target shapes have different numbers of vertices, a challenge for many geometric methods.

Key Results

  • Reconstruction Accuracy: OperatorNet, especially when using both intrinsic and extrinsic shape differences, achieves the lowest reconstruction error (measured by $d_R$) compared to geometric baselines like SfL [5] and FuncChar [10]. For instance, Op.Net (Int+Ext) achieves a $d_R$ of 1.11 (x10⁻⁴), significantly better than SfL's 48.8 or FuncChar's 28.4.
  • Shape Interpolation: OperatorNet produces visibly smoother and less distorted interpolations compared to PointNet autoencoders and nearest neighbor retrieval, as shown in Figure 1 and Figure 9.
  • Shape Analogy: The method generates more natural and semantically meaningful analogies, correctly transferring attributes like gender and pose, as demonstrated in Figures 10 and 13.
  • Handling Different Discretizations: OperatorNet successfully reconstructs shapes with different vertex counts than the base shape, a feat difficult for geometric approaches (Figure 7 and Figure 12).
  • Ablation Studies: Performance is best when using all three types of shape differences (Area, Extrinsic, Conformal), and shallower encoder architectures tend to perform better.

Limitations and Future Work

  • Mesh-Based Representation: The current framework is primarily designed for shapes represented as triangle meshes.
  • Future Directions: The authors suggest extending the framework to:
  • Learn optimal inner products from data.
  • Adapt the pipeline to other shape representations, such as point clouds or triangle soups.

Summary updated

Abstract

This paper proposes a learning-based framework for reconstructing 3D shapes from functional operators, compactly encoded as small-sized matrices. To this end we introduce a novel neural architecture, called OperatorNet, which takes as input a set of linear operators representing a shape and produces its 3D embedding. We demonstrate that this approach significantly outperforms previous purely geometric methods for the same problem. Furthermore, we introduce a novel functional operator, which encodes the extrinsic or pose-dependent shape information, and thus complements purely intrinsic pose-oblivious operators, such as the classical Laplacian. Coupled with this novel operator, our reconstruction network achieves very high reconstruction accuracy, even in the presence of incomplete information about a shape, given a soft or functional map expressed in a reduced basis. Finally, we demonstrate that the multiplicative functional algebra enjoyed by these operators can be used to synthesize entirely new unseen shapes, in the context of shape interpolation and shape analogy applications.
BibTeX
@inproceedings{huang2019operatornet,
  author = {Ruqi Huang and Marie-Julie Rakotosaona and Panos Achlioptas and Leonidas J. Guibas and Maks Ovsjanikov},
  title = {{OperatorNet: Recovering 3D Shapes From Difference Operators}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00868},
}

Loading related content...

Figure