Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

Wang Yifan, Lukas Rahmann, Olga Sorkine-Hornung

TL;DR

This paper introduces Implicit Displacement Fields (IDF), a novel 3D representation that decomposes complex shapes into a smooth base surface and a high-frequency displacement field. By leveraging the frequency biases of neural networks, IDF achieves unsupervised disentanglement of geometric details, leading to superior reconstruction quality, better training stability, and the ability to transfer fine details between different 3D models.

What Problem Does This Paper Address?

Neural implicit functions (like Signed Distance Functions or SDFs) have become a go-to for continuous 3D representation. However, they face a "spectral bias" dilemma: standard networks naturally learn low-frequency, smooth shapes first. To capture fine details (high-frequency signals), researchers often turn to spatial partitioning (which consumes massive memory) or specialized architectures like SIRENs. While SIRENs can represent details, they are notoriously difficult to train, often prone to overfitting or getting stuck in poor local minima.

The authors seek a middle ground: a representation that is as detailed as a high-frequency SIREN but as stable and easy to optimize as a smooth, low-frequency network.

Methods

The core idea is inspired by displacement mapping, a classic computer graphics technique where a detailed surface is created by offsetting a smooth base surface along its normals (see Figure 1).

Figure 1: Displacement mapping in 1D. The detailed surface (upper blue) is created by offsetting samples of the base surface (upper black) using the height map shown below.

The authors translate this into a neural framework using two SIREN networks:

  1. Base Network ($N^{\omega_B}$): A low-frequency network that represents a smooth "base" SDF.
  2. Displacement Network ($N^{\omega_D}$): A high-frequency network that predicts the scalar displacement distance along the base surface's normal.
Figure 2: Method overview. We represent detailed geometries as a sum of a coarse base shape represented as low-frequency signed distance function and a high-frequency implicit displacement field, which offsets the base iso-surface along the base’s normal directions.

Key Technical Innovations: * Inverse Displacement Formulation: To make training efficient, they use an "inverse" displacement field. This allows them to pre-sample points on the ground-truth detailed surface and map them back to the base surface, avoiding costly dynamic sampling during optimization. * Progressive Training: They employ a coarse-to-fine training strategy, first fitting the smooth base shape and then gradually "turning on" the displacement network. * Transferable IDF: By replacing raw 3D coordinates with scale-and-translation invariant features relative to the base surface (using a global context descriptor and FiLM conditioning), they allow the learned displacement "texture" to be applied to entirely new base shapes.

Figure 6: Illustrations for transferable and non-transferable implicit fields. The transferable modules take scale-and-translation invariant features as inputs, describing the relative position of the query point to the base shape.

Datasets

The authors evaluated their method on:

  • Sketchfab-16: A set of 14 high-resolution 3D models from Sketchfab and 2 from the Stanford 3DScanRepo.
  • Detail Transfer Sets: Shapes provided by Berkiten et al. (2017), Yang et al. (2020), and Zhou & Jacobson (2016) were used to test how well details (like clothing wrinkles or facial features) could be transplanted.

Key Insights and Ideas

  • Frequency Disentanglement by Construction: By simply setting different frequency hyperparameters ($\omega$) for the two networks, the model naturally partitions the geometry into "structure" and "detail" without needing explicit labels.
  • Geometric Constraints as Regularizers: Forcing the high-frequency signal to follow the normal of a smooth base surface acts as a powerful geometric prior. This prevents the "noisy" artifacts often seen when training high-frequency networks from scratch.
  • Relative vs. Absolute Coordinates: The shift from absolute $XYZ$ coordinates to relative features (distance to surface, local context) is what unlocks transferability, effectively creating a "neural height map" that doesn't care about the underlying global pose.

Key Results

  • State-of-the-Art Fidelity: IDF outperformed all baselines in reconstruction accuracy. On the Sketchfab-16 dataset, it achieved a Chamfer distance of $1.22 \times 10^{-3}$, significantly lower than FFN ($5.47$) or D-SDF ($2.85$).
  • Efficiency: While NGLOD (a spatial partitioning method) achieved comparable results at its highest level of detail, it required 256x more parameters and nearly 200x more memory (946MB vs. 4.8MB for IDF).
  • Stability: Unlike standard high-frequency SIRENs, which often diverged during training, IDF remained stable across all tested models.
  • Detail Transfer: The method successfully transferred complex, spatially-varying details (like eyebrows or fabric folds) to new target shapes without requiring any correspondence mapping between the source and target.

Limitations and Future Work

  • Pre-alignment Requirement: For detail transfer, the source and target shapes currently need to be roughly aligned. The authors suggest exploring sparse correspondences in the future to automate this alignment.
  • Sparse Data Sensitivity: Stress tests showed that when training data is extremely sparse (e.g., only 1% of points), the model can overfit and generate high-frequency noise.
  • Inference Speed: While memory-efficient, the current implementation's inference time is slightly slower than some optimized spatial structures like NGLOD, leaving room for performance optimization.

Summary updated

Abstract

We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high frequency signal is constrained geometrically by the low frequency signal. Importantly, this disentanglement is unsupervised thanks to a tailored architectural design that has an innate frequency hierarchy by construction. We explore implicit displacement field surface reconstruction and detail transfer and demonstrate superior representational power, training stability and generalizability.
BibTeX
@inproceedings{yifan2022geometry,
  author = {Wang Yifan and Lukas Rahmann and Olga Sorkine-Hornung},
  title = {{Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}
Figure