Localized Generative Editing for Neural 3D Scenes

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2104 papers140 conceptsfrozen 2026-10-04

When you ask a generative model to "put a chef’s hat on a cat" in a 2D image, the model often redraws the entire cat, changing its breed, pose, or color. In 3D, this problem—often called global drift or semantic leakage—is even more destructive. Because implicit representations like Neural Radiance Fields (NeRFs) store scene information in global network weights, a small edit to a single object can cause the entire background to warp, blur, or shift.

To solve this, researchers have moved away from global optimization toward localized editing. The goal is to ensure that modifications are mathematically and visually confined to a specific region of interest, leaving the rest of the 3D world "bit-perfectly" intact.

The Challenge of Global Coupling

In traditional 3D engines, objects are discrete meshes. If you move a chair, the floor remains unchanged. However, as Minghao Chen and colleagues note in DGE, implicit 3D representations lack this spatial discretization. When we use a diffusion model to guide a NeRF or Gaussian Splatting (3DGS) scene toward a new text prompt, the gradients propagate through the entire structure.

This coupling leads to several failure modes:

  1. Background Corruption: The environment loses its original texture as the model tries to satisfy the global prompt.
  2. Identity Loss: Modifying a specific facial feature might inadvertently change the subject's identity, a problem addressed by Yunjie Wu in FaceG2E.
  3. Detail Bleeding: Generative hallucinations intended for a new object "leak" into the surrounding space, as described in GSEditPro.

Spatial Masking and Volumetric Blending

The most direct solution is to define a 3D "safe zone." Ori Gordon and the authors of Blended-NeRF pioneered this by restricting scene updates to a specific 3D region of interest. By blending a frozen, pretrained NeRF (the background) with a learnable "editable" NeRF (the object), they ensure that the background remains mathematically identical to the original.

This spatial partitioning is often achieved by:

  • Lifting 2D Masks: Using models like SAM to segment an object in 2D views and projecting those masks into 3D space to create a volumetric editing boundary, a technique used in 3D-LATTE and 3DEgo.
  • Grid Partitioning: In TetGS, Hanxi Liu embeds Gaussians within a tetrahedral grid, freezing the parameters of the "preserved" region while allowing the "target" region to deform and re-texture.

Latent Space Surgery

Rather than optimizing raw 3D primitives, some methods operate within the latent space of 3D generators. The intuition here is that if we can find the specific "tokens" or "latents" that correspond to a physical part of an object, we can edit them in isolation.

Gengxiong Zhuang introduces this in VoxHammer through "contextual feature replacement." By identifying which 3D latent features contribute to a specific region, the model can swap them out for new features while keeping the surrounding latent context frozen. Similarly, Teng-Fang Hsiao in VecSet-Edit discovered that unordered latent tokens in Large Reconstruction Models (LRMs) correspond to specific geometric regions, allowing for "surgical" edits without retraining.

Compositional and Part-Based Representations

A more fundamental shift involves moving away from "monolithic" 3D models. If an object is built from the ground up as a collection of independent parts, localization becomes trivial.

Despoina Paschalidou and the creators of PartNeRF represent shapes as a set of local NeRFs. This allows a user to move, recolor, or swap a "chair leg" without the "chair back" ever knowing a change occurred. Lihe Ding takes this further in FullPart by assigning each part its own high-resolution voxel grid, enabling part-level inpainting where changes are strictly confined to a specific bounding box.

Surgical Grafting and Merging

The final stage of localized editing is the "graft." Once a part has been modified or a new object generated, it must be integrated back into the original scene.

Junliang Ye in Nano3D treats editing as a grafting task. Instead of re-rendering the whole scene, the system identifies changed voxels through connectivity analysis and "sews" them into the original asset. Chaoyang Wang uses a similar "segment and merge" pipeline in PrEditor3D, ensuring that only the targeted 3D mesh faces are updated, preserving the original geometry with bit-perfect accuracy.

By combining these spatial constraints with generative priors, the field is moving toward a future where 3D scenes are as editable as layers in Photoshop, allowing for complex, multi-entity updates that respect the integrity of the reconstructed world.

Papers in this version

Showing 60 of 104 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 99 papers
  2. Version 1 · 2026-10-04 · 104 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_localized_3d_scene_editing_3d55dd14,
  title = {localized 3D scene editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localized-3d-scene-editing/rdr_3d55dd14069a4e50ae19b367885b2ebb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_localized_generative_editing_for_neural__3d55dd14_v1,
  title = {Localized Generative Editing for Neural 3D Scenes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localized-generative-editing-for-neural-3d-scenes/rdr_3d55dd14069a4e50ae19b367885b2ebb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}