Editing Neural Implicit Representations

154 concepts118 papers1 proposal

Neural implicit representations, such as Neural Radiance Fields (NeRFs) and coordinate-based Multi-Layer Perceptrons (MLPs), have fundamentally transformed 3D reconstruction. By mapping continuous 3D coordinates to scene properties like density, color, or signed distance values, these models achieve exceptional photorealism and resolution independence.

However, as detailed in the survey State of the Art on Neural Rendering, a profound gap remains between the abstract weight space of a neural network and the intuitive handles required by human creators. Traditional computer graphics rely on explicit primitives—vertices, edges, polygons, and UV texture maps—that provide direct spatial control. In contrast, an implicit representation stores the entire scene globally within network weights. This creates a "black box" where isolating a single object, changing a surface color, or animating a joint becomes a complex optimization challenge rather than a simple click-and-drag operation.

The Core Challenge: Global Entanglement

To understand why editing an implicit field is difficult, consider what happens when you modify a standard MLP-based NeRF. Because every weight in a global network can potentially influence any point in the 3D space, the representation suffers from global entanglement.

As highlighted by Wudi Chen et al. in KaRF, attempting to alter a localized feature (such as the color of a single object) often triggers catastrophic interference, causing unrelated parts of the scene to shift or discolor unpredictably.

Furthermore, geometry and appearance are tightly coupled within these weights. Chong Bao demonstrates in NeuMesh that this entanglement prevents fine-grained editing; an artist cannot easily repaint a surface without accidentally altering its underlying geometric structure, leading to artifacts like ghosting or shape distortion.

To overcome this rigidity, modern research focuses on three main strategies: anchoring implicit fields to explicit structures, disentangling geometry from appearance, and isolating deformations through localized fields.

Strategy 1: Hybrid Representations and Explicit Cages

The most direct way to introduce editability is to marry the photorealism of implicit fields with the structural handles of explicit graphics primitives. By partitioning space or wrapping the neural field in an explicit geometric "container," edits can be localized.

Global MLP (Entangled)  -->  Hybrid Voxel/Mesh Cage (Localized)
   [ All Weights ]              [Voxel A] -> Local Neural Field
      |     |                   [Voxel B] -> Local Neural Field
   (Global Changes)            (Edits contained within Voxel A)

An early realization of this concept was developed by Lingjie Liu et al. in Neural Sparse Voxel Fields. Instead of a single global network, the scene is organized into a sparse voxel octree where each voxel contains its own local neural features. This explicit spatial partitioning allows users to directly select, translate, duplicate, or delete specific voxels without retraining the rest of the scene.

For complex physical interactions like deformations or fractures, the authors of Neural Impostor proposed wrapping a volumetric radiance field inside an explicit tetrahedral mesh "cage." By using barycentric coordinates to anchor the continuous neural field to the discrete vertices of the cage, the underlying volumetric data warps naturally alongside the mesh. This allows standard physics engines to manipulate complex neural assets (like fur or hair) as if they were traditional assets.

Recently, this hybrid paradigm has shifted toward point-based explicit primitives, notably 3D Gaussian Splatting (3DGS). As shown by Florian Barthel et al. in Gaussian Splatting Decoder, converting implicit tri-plane representations into explicit 3D Gaussians provides immediate structural handles, enabling generative assets to be exported directly into standard engines like Blender or Unity for localized manipulation.

Strategy 2: Disentangling Geometry from Appearance

In traditional digital content creation, an object's shape (mesh) and its surface appearance (UV texture map) are completely separate. Replicating this separation within a continuous neural volume is essential for tasks like texture painting or re-lighting.

To achieve this, Fanbo Xiang et al. introduced NeuTex, which maps 3D spatial coordinates within a neural volume to a 2D texture coordinate space (UV space) via a learnable mapping network. When a ray is marched through the volume, its appearance is sampled from this explicit 2D neural texture map rather than being predicted directly by a spatial MLP. This allows human artists to paint logos or swap textures on a 2D canvas and see those changes map accurately onto the 3D neural volume.

3D Ray Point (x,y,z)  -->  [ Mapping Network ]  -->  2D UV Coordinate (u,v)
                                                          |
                                                   [ 2D Texture Map ]
                                                          |
                                                   (Editable Canvas)

This concept is extended to dynamic, high-fidelity human avatars in Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization. Here, Connor Z. Lin et al. combine implicit functions for fine geometric details (like wrinkles) with explicit UV texture parameterization, ensuring that facial expressions can change while the underlying skin texture remains anchored and editable.

For rapid color manipulation, Alessio Mazzucchelli developed IReNe, which isolates appearance by targeting only the "diffuse" neurons in a network's final layer. By decoupling the view-dependent specular components from the base diffuse color, the framework allows users to execute localized color updates in seconds without causing color bleeding across object boundaries.

Strategy 3: Localized Deformation Fields

When a scene undergoes non-rigid motion—such as a face changing expressions or a hand bending—modeling the transformation globally introduces severe distortion. Instead, researchers use deformation fields to map coordinates from a "deformed space" back to a static, highly detailed "canonical space."

Deformed Space (x',y',z') --> [ Local Deformation Fields ] --> Canonical Space (x,y,z)
                                    |                                |
                             (Landmark-based)                (Static High-Detail)

In Implicit Neural Head Synthesis via Controllable Local Deformation Fields, Chuhan Chen et al. address the limitations of global deformation networks by breaking them down into multiple localized deformation fields centered around specific facial landmarks. This localization ensures that moving the left eyelid, for instance, only affects the weights governing that specific spatial region, enabling precise, asymmetric animations like winking.

For direct geometric sculpting without explicit meshes, Arturs Berzins et al. proposed a different mathematical approach in Neural Implicit Shape Editing using Boundary Sensitivity. Instead of warping space, they derive a boundary sensitivity formula that relates a change in the network's internal parameters directly to the physical movement of the implicit surface boundary. This allows users to apply localized forces or volume-preserving constraints directly to the implicit shape, bypassing mesh conversion entirely while maintaining mathematical precision.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 133 papers
  2. Version 1 · 2026-10-04 · 203 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_representation_editability_028e4bb0,
  title = {implicit representation editability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-representation-editability/rdr_028e4bb09b05473799068a543331e3bc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_implicit_representation_editability_028e4bb0_v2,
  title = {implicit representation editability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-representation-editability/rdr_028e4bb09b05473799068a543331e3bc/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake