Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

Yifan Wang, Lukas Rahmann, Olga Sorkine-Hornung

Introduction

Neural implicit functions, such as Signed Distance Functions (SDFs), have revolutionized 3D geometry by providing continuous, resolution-independent representations. However, they frequently encounter a "spectral bias" dilemma: standard networks struggle to capture fine, high-frequency details like wrinkles or sharp edges, while networks specifically designed for high frequencies (like SIRENs) are notoriously difficult to train, prone to overfitting, and often fall into local minima.

Existing solutions typically rely on spatial partitioning, such as octrees, which unfortunately ties memory usage directly to geometric complexity. To address this gap, researchers have introduced Implicit Displacement Fields (IDF). This novel neural representation decomposes complex 3D shapes into a smooth base surface and a high-frequency displacement field, offering a geometrically principled approach that is easy to optimize without massive memory overhead.

Methodology: Neural Displacement Mapping

The core concept of IDF is inspired by traditional displacement mapping in computer graphics, where a smooth mesh's vertices are moved along their normals based on a 2D texture map. The authors elegantly translate this into the neural implicit domain.

Decomposing a complex 3D shape into a base SDF and an implicit displacement field.

The Decomposition: A shape is represented as the sum of a low-frequency base SDF ($f$) and a high-frequency implicit displacement field ($d$). The detailed surface is defined by offsetting the base surface strictly along its normal direction: $\hat{f}(x) = f(x + d(x)n)$.

Frequency Hierarchy: To achieve this, the architecture employs two SIREN networks with distinct frequency hyperparameters ($\omega$). A "slow" network ($\omega_B = 15$) learns the smooth base shape, while a "fast" network ($\omega_D = 60$) is dedicated to learning the fine, high-frequency details.

Illustration of the implicit displacement and the inverse displacement mapping used for efficient training.

Inverse Mapping for Efficiency: Calculating the displacement dynamically during training can be computationally prohibitive. To circumvent the cost of dynamic sampling, the authors formulate an inverse displacement field. This allows the model to use pre-sampled points directly from the ground-truth detailed surface, significantly streamlining the optimization process.

Key Insights and Innovations

  • Normal-Aligned Constraints: By restricting the displacement to occur only along the normal of the base surface, the model remains "geometry-consistent." This crucial constraint prevents the high-frequency network from hallucinating floating artifacts or generating geometry far from the actual surface.
  • Unsupervised Disentanglement: Thanks to the architectural constraints and a progressive training scheme (training the base first, followed by the details), the network naturally learns to separate "shape" from "texture" without requiring labeled data.
  • Attenuation Functions: The model introduces a spatial attenuation function ($\chi$) that actively suppresses the high-frequency displacement network in "void" regions far from the surface. This dramatically improves training stability and reduces unwanted noise.
  • Transferability: By replacing raw 3D coordinates with scale-and-translation invariant features, the learned "displacement style" (e.g., a specific knit pattern) can be transferred to a completely different base shape, enabling robust geometric texture transfer.

Experimental Results

The researchers evaluated IDF on 16 high-resolution 3D shapes sourced from Sketchfab and the Stanford 3DScanRepo (including complex models like the Asian Dragon), as well as specialized datasets for faces and clothing.

  • Superior Fidelity: IDF outperformed state-of-the-art methods in reconstruction accuracy. On the SketchFab-16 benchmark, IDF achieved a Chamfer distance of 1.22, compared to 1.35 for NGLOD-LOD6 and 2.85 for D-SDF.
  • Parameter Efficiency: While NGLOD-LOD6 produces comparable visual results, it requires 300x more parameters than the IDF model. IDF proves to be a highly lightweight alternative that does not sacrifice geometric detail.
  • Training Stability: Unlike standard high-frequency SIRENs, which often diverge on complex shapes, IDF's progressive training scheme remained highly stable across all test cases.

Limitations and Future Directions

The primary limitation of the current IDF model is the requirement for spatial alignment during detail transfer; the source and target shapes must be roughly aligned in 3D space for the feature extractor to function correctly.

Future work aims to integrate sparse correspondences to enable automatic shape alignment. Additionally, researchers plan to optimize inference speed (currently ~250 seconds for a $512^3$ grid) by exploiting the mathematical property that SIREN derivatives are themselves SIRENs, which could bypass expensive automatic differentiation steps for normal calculations.

Summary updated

Abstract

We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high frequency signal is constrained geometrically by the low frequency signal. Importantly, this disentanglement is unsupervised thanks to a tailored architectural design that has an innate frequency hierarchy by construction. We explore implicit displacement field surface reconstruction and detail transfer and demonstrate superior representational power, training stability and generalizability.
BibTeX
@inproceedings{wang2022geometry,
  author = {Yifan Wang and Lukas Rahmann and Olga Sorkine-Hornung},
  title = {{Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}
Figure