When we think about representing 3D shapes digitally, our minds typically gravitate toward explicit representations. We imagine voxel grids (the 3D equivalent of pixels), point clouds (collections of discrete coordinates), or polygon meshes (networks of vertices and faces).
While intuitive, these explicit representations suffer from severe trade-offs. Voxel grids scale cubically in memory ($O(N^3)$), forcing a painful choice between low resolution and high memory consumption. Point clouds lack explicit surface topology, leaving holes and disconnected geometry. Polygon meshes can represent sharp details efficiently, but optimizing them via deep learning is notoriously difficult because changing the topology—such as turning a solid sphere into a donut with a hole—requires tearing and stitching the underlying graph structure.
Neural implicit surface representations offer a paradigm shift. Instead of storing the geometry explicitly, we train a coordinate-based neural network to represent the scene as a continuous mathematical field.
Explicit (Voxel Grid) Implicit (Neural Field)
┌───┬───┬───┐ ┌───────────────────┐
│ 0 │ 0 │ 0 │ │ │
├───┼───┼───┤ (x,y,z) │ Continuous │
│ 0 │ 1 │ 0 │ ───────────► │ Neural Network │ ──► Scalar Value
├───┼───┼───┤ │ (MLP) │
│ 0 │ 0 │ 0 │ │ │
└───┴───┴───┘ └───────────────────┘
Memory scales cubically Memory scales with network capacity
The Coordinate-Based Architecture
The core mechanism is elegant in its simplicity. We parameterize a 3D shape using a Multi-Layer Perceptron (MLP). The network takes a single 3D spatial coordinate $x = (x, y, z)$ as its input and outputs a scalar value describing the geometric state at that exact point.
Because the input space is continuous, this representation has theoretically infinite resolution. We can query the network at any arbitrary precision during inference, completely bypassing the discrete grid constraints of traditional volumetric methods.
Depending on how we define the network's scalar output, neural implicit representations generally fall into two categories: occupancy fields and signed distance functions.
Occupancy Networks
In Occupancy Networks: Learning 3D Reconstruction in Function Space, Lars M. Mescheder et al. framed 3D reconstruction as a continuous binary classification problem. The neural network predicts the probability that a given coordinate lies inside or outside the object:
$$f(x) \in [0, 1]$$
The actual 3D surface is implicitly defined as an isosurface where the probability equals a specific threshold, typically $f(x) = 0.5$. This formulation allows the network to represent arbitrary topologies and smooth boundaries without requiring a massive memory footprint.
Signed Distance Functions (SDFs)
An alternative, highly popular formulation is the Signed Distance Function, popularized in the context of deep learning by frameworks like DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. Instead of a binary classification, the network regresses a continuous scalar representing the exact distance from the query point to the nearest surface boundary:
$$f(x) = d$$
The sign of the output indicates whether the point is inside (negative) or outside (positive) the object. Under this formulation, the physical surface of the 3D object is defined precisely as the zero-level set:
$$\mathcal{S} = \{ x \in \mathbb{R}^3 \mid f(x) = 0 \}$$
To extract a traditional polygon mesh for rendering or simulation, algorithms like Marching Cubes or Dual Contouring are run over the continuous field to find this zero-crossing boundary.
Why It Works: Gradients and the Eikonal Constraint
The true superpower of neural implicit surfaces lies in differentiability. Because the geometry is encoded within a neural network, we can compute analytical gradients of the surface representation with respect to the spatial coordinates using backpropagation.
For a valid Signed Distance Function, the magnitude of its spatial gradient must satisfy the Eikonal equation:
─────────── $$\|\nabla_x f(x)\| = 1$$ ───────────
This mathematical constraint states that if you move one unit in space, the distance to the surface must change by exactly one unit. To enforce this behavior when training directly from raw, unorganized data (like point clouds), researchers apply an Eikonal loss term. For instance, Daiwen Sun et al. in DSR: Dynamical Surface Representation as Implicit Neural Networks for Protein utilize this gradient penalty to force the network to behave as a valid distance metric across the entire 3D volume, preventing the network from collapsing or producing noisy artifacts.
Furthermore, the spatial gradient evaluated at the zero-level set gives the exact, continuous surface normal vector:
$$n = \frac{\nabla_x f(x)}{\|\nabla_x f(x)\|}$$
Knowing the exact surface normal is crucial for physical simulations, calculating lighting equations, and aligning shapes. In Controlling Neural Level Sets, Matan Atzmon et al. demonstrated that by directly optimizing the zero-level set against raw, noisy point clouds, the network naturally learns to interpolate missing data and complete complex geometries gracefully.
Conditioning and Generalization
An isolated MLP can only encode a single 3D shape. To make neural implicit representations useful for general machine learning tasks, we must condition the network on external observations, such as images, sparse point clouds, or category embeddings.
This is typically achieved by passing a latent shape code $z$ alongside the spatial coordinate:
$$f(x, z) = d$$
Image/Sensor Input ──► Encoder ──► Latent Code (z) ┐
├─► Decoder MLP ──► Distance/Occupancy
Spatial Coordinate (x,y,z) ┘
For example, Qiangeng Xu et al. developed DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction, which extracts both global image features and local point-to-pixel projections to condition the MLP. This allows the network to capture fine-grained details, like thin chair legs or holes, from a single 2D image.
When dealing with broader categories, Zixuan Huang et al. showed in Planes vs. Chairs: Category-Guided 3D Shape Learning Without any 3D Cues that a hypernetwork can be trained to predict the weights of the shape-representing MLP itself, allowing zero-shot generalization across dozens of distinct object categories without explicit 3D supervision.
Applications Across Modalities
Because implicit fields provide continuous, gradient-based feedback, they have become a foundational tool for multi-modal fusion and robotics:
- Tracking and Pose Estimation: In iCaps: Iterative Category-level Object Pose and Shape Estimation, Xinke Deng et al. leveraged the continuous nature of SDFs to perform gradient-descent alignment between observed depth maps and implicit shapes, refining both object pose and shape iteratively.
- Sensory Fusion: Dejan Azinovic demonstrates in Neural RGB-D Surface Reconstruction how Truncated Signed Distance Functions (TSDFs) can be optimized alongside color and depth inputs to generate clean, metric-accurate room meshes that surpass standard volumetric density outputs.
- Robotic Interaction: Beyond vision, Jayjun Lee et al. introduced ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation, utilizing implicit fields to map out external contact probabilities and object boundaries simultaneously by fusing visual data with tactile shear fields. Similarly, Mauro Comi et al. showed in TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing that a robot can reconstruct smooth, continuous 3D shapes of completely unseen objects purely through a sequence of localized tactile "pokes."
By shifting the representation of 3D structure from discrete spatial bins to continuous neural functions, implicit surfaces provide a mathematically rigorous, memory-efficient, and fully differentiable framework that bridges the gap between deep learning and the physical world.
Key Foundational Work
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018) — contributed 1 concept elements to this direction.