Local Appearance Editing of Dynamic Gaussian Scenes in Seconds

A dynamic 3D scene reconstructed from video (a person cooking, a dog walking across a room) can now be replayed from new camera angles in real time. Editing such a scene is still slow. Suppose you want to recolor the dog's collar or put a pattern on a mug. Today's methods run a 2D image editor on many rendered frames and then re-optimize the 3D model for 40 minutes to 2 hours. Our observation is that for the most common kind of edit, a local change in appearance that leaves shape and motion alone, most of this work is unnecessary. The reconstructed scene already records which 3D element appears at every pixel of every camera at every moment. That record gives exact matches between views and across time, with no optical-flow estimation. And once geometry is frozen, choosing new colors for a chosen set of elements is a linear least-squares problem that can be solved directly instead of by hours of gradient descent.

We propose a direct-fitting editor for dynamic Gaussian scenes. The scene is represented with 4D Gaussian Splatting: a fixed "canonical" cloud of small colored 3D blobs, plus a learned deformation field that moves and reshapes them over time. The editor does four things. It uses the model's own canonical coordinates to link pixels across cameras and times. It picks a small set of camera-time keyframes that together see every Gaussian in the target region. It makes the 2D edits of those keyframes agree with one another using those links. Finally, it solves for new colors of only the selected Gaussians in closed form. Temporal consistency follows from the representation itself: canonical colors do not depend on time, so the unchanged deformation field carries the edit through the whole sequence. Background preservation also follows from the representation, because background parameters are not unknowns in the solve. The target is edit turnaround under one minute, against 40 minutes for Instruct-4DGS and about 60 minutes for CTRL-D.

Core Claim

For appearance-only local edits, dynamic scene editing does not need score distillation, iterative dataset updates, or optical flow. It needs three things: correspondences read out of the reconstruction itself, keyframes chosen so that every target Gaussian is seen, and a linear solve over a masked subset of Gaussians. The claim is falsifiable in two ways. It fails if the method does not match the edit quality and temporal consistency of 40–60 minute optimization pipelines. It also fails if a simpler variant does just as well, namely editing only the $t=0$ views and fitting colors by gradient descent.

How This Relates to Prior Work

The static-scene result that motivates this work is DGE. DGE edits a few key views of a static Gaussian scene, pushes features to the other views along epipolar lines, and fits the result directly. Editing time drops to about 4 minutes, against about 51 for Instruct-NeRF2NeRF. Epipolar lines only describe correspondence in a rigid, unmoving scene, however, and DGE states that it cannot handle motion. We keep DGE's structure: make the 2D views consistent first, then fit once. We replace epipolar geometry with canonical-coordinate correspondence, which remains valid under non-rigid motion.

Three dynamic editors define the latency and quality bar, and each works around the problem differently. Instruct-4DGS edits the canonical Gaussians from multi-view renderings at $t=0$. It then needs score-based temporal refinement to remove "motion artifacts", which brings the total to 40 minutes. CTRL-D personalizes an image editor on one edited pair and co-optimizes the canonical Gaussians and the deformation field in about 60 minutes. 4DGS-Craft adds geometry-aware features and a Gaussian selector inside an iterative loop.

Instruct-4DGS matters most for us because its failure is the thing we need to explain. Our hypothesis is that its $t=0$ artifacts come from two sources. The first is coverage: Gaussians hidden at $t=0$ but visible later receive no signal. The second is residual inconsistency between views. Neither is inherent to editing canonical appearance. If that hypothesis is wrong, the proposal fails, and the ablations below are designed to test it.

Two training-free methods constrain the design. Dynamic-eDiTor and Instruct 4D-to-4D enforce consistency with sub-grid attention or anchor attention, and both propagate edits with RAFT optical flow. Monocular Dynamic View Synthesis: A Reality Check shows that tracking on realistic motion loses about 30% in keypoint-transfer accuracy (PCK-T). That is the reason we do not rely on 2D flow. Dynamic-eDiTor also has no way to confine an edit to one object.

Two further papers supply parts of the method. The backbone is 4D-GS: canonical Gaussians plus a HexPlane deformation field that predicts position, rotation, and scale offsets but leaves color unchanged. This is the property that makes canonical color edits consistent over time. SC-GS, which drives Gaussians with sparse control points, is a second backbone for testing whether the idea transfers. The idea of detecting which region changed by comparing before and after, rather than asking for a mask, is adapted from the voxel XOR merge in Nano3D.

Research Program

Correspondence from the reconstruction. For any camera $v$ and time $t$, we render a map that stores, at each pixel, the canonical 3D position of the Gaussians that contribute to it, weighted by their compositing weights. We also render the IDs of the dominant Gaussians. Two pixels at different camera-time pairs correspond when their dominant canonical positions agree and both pass a depth-visibility test. This works for any pair of views and times, including large time gaps where flow fails. It costs one extra rasterization pass per frame, run at the backbone's 82 FPS.

Selection. The user marks the target in one rendered frame, either by clicking or with a text-prompted 2D segmenter. The ID map lifts that mark to a set $S$ of canonical Gaussians. The set is then cleaned by voting across a few other camera-time frames, which removes Gaussians that only occasionally fall inside the mask. In a mask-free variant, the key frames are edited without a mask first, and $S$ is taken to be the Gaussians whose pixels changed consistently across frames. This is the canonical-space analogue of Nano3D's XOR merge.

Coverage-driven keyframes. Each Gaussian in $S$ must receive a real color signal. We therefore choose keyframes greedily over (camera, time) pairs. The greedy step adds whichever frame raises the total visibility weight of $S$ the most, subject to a cap of $K$ frames. We expect $K$ to be roughly 8 to 16. Instruct-4DGS is the special case in which every keyframe has $t=0$.

Consistent keyframe edits. An instruction-following image editor (InstructPix2Pix) edits the $K$ keyframes. Following DGE, attention features from one anchor keyframe are injected into the others. The injection is placed at corresponding pixels given by the canonical maps, not along epipolar lines.

Closed-form color solve. With geometry and deformation frozen, the color of pixel $p$ is $C(p)=\sum_i w_i(p)\, c_i(d_p)$. Here $w_i(p)$ is Gaussian $i$'s fixed compositing weight at that pixel and $c_i(d_p)$ is its spherical-harmonic color toward the viewing direction $d_p$. This color is linear in the coefficients. We solve

$$\min_{\{c_i\}_{i\in S\cup B}} \sum_{p\in\mathcal{K}} \big\| C(p) - I_{\text{edit}}(p) \big\|^2 + \lambda \sum_{i\in B} \| c_i - c_i^{0} \|^2 ,$$

where $\mathcal{K}$ is the set of keyframe pixels inside the edited region and a thin surrounding band, and $I_{\text{edit}}$ is the edited keyframe image. The set $B$ is a one-ring band of boundary Gaussians, which are pulled toward their original colors $c_i^0$ with strength $\lambda$. The band is there to soften seams. All other Gaussians stay fixed. The problem is sparse and is solved with conjugate gradient on the GPU. Because no Gaussians are added, rendering speed after the edit is unchanged.

Scope. The method is limited to texture, color, and style edits of existing surfaces. It does not add objects or change shape or motion. We state this scope openly. Instruct-4DGS has the same scope, and it is also the part of CTRL-D's example edits that does not add new volume.

Baselines

The naive reference edits each frame independently with InstructPix2Pix and then retrains 4D-GS on the edited frames. The other methods should improve on it, and it shows how large the consistency problem is on each scene.

The strongest direct competitor on time and scope is Instruct-4DGS, which takes 40 minutes on a single GPU. It solves the same appearance-only problem on the same 4D-GS backbone. It runs on the same pretrained 4D-GS models, the same instructions, and the same masks for partial edits, with its default schedule. Beating it means matching or exceeding its CLIP directional similarity and temporal consistency, with background preservation at least as good, in under one minute of wall-clock time on the same GPU.

Dynamic-eDiTor reports a CLIP directional similarity of 0.1849 across DyNeRF and DyCheck. That is the quality number on the shared benchmarks we have to reach. Because it cannot localize, we compare it in two forms: as published, and with its output composited through our selection mask. This keeps it from being penalized only for touching the background.

4DGS-Craft reports a VBench subject-consistency score of 0.942 on DyCheck "pillow", against 0.931 for CTRL-D. We report the same metric on that scene. Instruct 4D-to-4D, at about 2 hours, is included where code runs on our hardware.

Four ablations isolate the key ingredients. The first restricts keyframes to $t=0$, which isolates coverage. The second swaps canonical correspondence for RAFT flow, then for static epipolar injection as in DGE, and then removes correspondence entirely. The third replaces the closed-form solve with Adam on the same color unknowns, which isolates the solver from the formulation. The fourth removes the boundary band, which tests whether it controls seams.

Why not just…?

Why not just do what Instruct-4DGS's first stage does and skip the refinement? This is the real threat. Instruct-4DGS reports motion artifacts when only $t=0$ canonical editing is used. We attribute them to missing coverage and inconsistent views, not to anything fundamental. The $t=0$-keyframe ablation settles the question. The prediction is that edit quality at late timesteps will track the fraction of $S$'s visibility weight that is covered at $t=0$. If $t=0$ editing matches our method even on scenes with heavy occlusion, then coverage-driven keyframe selection contributes nothing, and the contribution shrinks to the solver and the correspondence mechanism.

Why not just propagate one edited keyframe with optical flow? Flow is estimated in 2D and breaks when surfaces become hidden or revealed. The Reality Check paper measures a roughly 30% PCK-T drop on realistic monocular motion. The RAFT-substitution ablation measures this failure directly on our edits, and we expect the gap to widen on DyCheck relative to N3DV.

Why not just fit colors by gradient descent instead of a linear solve? Gradient descent restricted to the color parameters would reach the same optimum. The only question is time. We expect the formulation to matter much more than the solver, and we will report both. If Adam on colors finishes within a few seconds of conjugate gradient, we will call the solver a convenience and not a contribution.

Why not just segment every frame with a 2D video segmenter? Split4D documents how per-frame mask errors turn into permanent 3D artifacts. Our selection lifts a single mark through Gaussian IDs and votes across frames, so it is never asked to agree with itself frame by frame. We measure boundary leakage under both selection schemes.

Proposed Experiments

Experiments run on N3DV/DyNeRF multi-view sequences and HyperNeRF scenes, where 4D-GS reconstructions are reliable. DyCheck monocular sequences serve as a stress test, because deformation there is known to be poor.

Edit quality is measured with CLIP directional similarity and a user preference study. Temporal and subject consistency are measured with VBench. Background preservation is measured as PSNR and LPIPS outside the target region against the original renders. Latency is split into correspondence, diffusion, and solve time, and we also report post-edit rendering FPS. All rendering is at held-out camera-time pairs that were never used as keyframes.

The first decisive check is the turnaround-quality comparison against Instruct-4DGS. The second is the $t=0$ coverage ablation described above.

The third check is a failure analysis on DyCheck. For each edit we compare local reconstruction error with local edit quality. The method's temporal consistency is only as good as the deformation field it inherits, so we expect edits to fail exactly where tracking fails. Showing this predicted failure pattern, rather than hiding it, is part of the result.

A final transfer run on SC-GS tests whether the approach depends on the specific 4D-GS backbone.