Picking Where to Spend the Diffusion Budget in 3D Scene Editing

Text-driven editing of a captured 3D scene ("make the bench wooden", "put a hat on the bear") is slow mainly because a 2D image-editing diffusion model has to run on many camera views. A diffusion model here means a generator that edits an image by repeatedly denoising it under a text prompt. Its outputs must then be fused back into the 3D representation. The fastest decoupled pipelines first edit views consistently and then fit the 3D scene once. Even so, they spend their diffusion calls in a fixed way: GaussCtrl edits every dataset view, DGE uses fixed keyframes along an orbit, and PrEditor3D uses four orthogonal views. None of them asks which few views would actually carry the edit. This note proposes to choose a small set of "anchor" viewpoints by maximizing an edit-weighted coverage score over the 3D primitives that the edit touches. The views are drawn not only from the captured cameras but also from a continuous space of virtual cameras. We then run the expensive multi-view diffusion edit only on those anchors.

Randomized greedy selection is what makes searching a continuous camera space practical. At each greedy step, instead of scoring every possible camera, we draw a fresh random batch of candidate poses and keep the best. Order statistics alone guarantee that a few dozen samples land in the top few percent of all poses, whatever the size of the space. Scoring a pose costs one cheap low-resolution visibility render. The whole selection therefore takes seconds, and the diffusion stage shrinks from hundreds of calls to roughly eight. The claim to test is that, at a matched wall-clock budget, coverage-chosen anchors give better localized edits and better preservation of untouched regions than uniform, keyframe, or all-view strategies in unbounded 360° scenes.

Core Claim

In decoupled editing, quality per second depends on where diffusion compute is spent, and that choice can be posed as monotone submodular maximization. Submodular means each added view helps less than the last. We work on a 3D Gaussian Splatting scene, which represents the scene as millions of small colored 3D Gaussians rendered by rasterization (3D Gaussian Splatting). Let $R$ be the Gaussians inside the edit region, with relevance weight $w_i$ for Gaussian $i$. Let $c_i(S)$ be the number of anchor views in set $S$ that see Gaussian $i$ unoccluded and at adequate projected size. The objective is

$$F(S) = \sum_{i \in R} w_i \, \phi\big(c_i(S)\big), \qquad \phi(c) = 1 - 2^{-c},$$

maximized over $|S| \le k$. The concave $\phi$ rewards seeing each edited Gaussian from a second and third angle but with diminishing returns. This matters because cross-view attention in GaussCtrl and DGE needs overlapping anchors to propagate a consistent appearance; pure set coverage would pick disjoint views. $F$ is a non-negative weighted sum of concave functions of counts, so it is monotone submodular. Greedy selection keeps its classical guarantee.

The randomized part is concrete. At each of the $k$ steps we draw $s$ fresh poses from a pose distribution fitted around the captured trajectory and add the one with the largest marginal gain. The chance that none of the $s$ draws falls in the top $\alpha$ fraction of poses is

$$(1-\alpha)^s,$$

which is about 0.046 for $s=60$ and $\alpha=0.05$. The pool is never enumerated, so its size does not enter the cost.

Relation to Prior Work

Three decoupled editors motivate the gap, and the method plugs into them. GaussCtrl edits all rendered views with depth-conditioned ControlNet and aligns them to a few reference views through cross-view attention, in about 9 minutes. Its references are chosen statically. DGE treats views as an orbital video, jointly edits fixed key views, and injects their features along epipolar lines into the other frames, in about 4 minutes. Its keyframing assumes a simple orbit. PrEditor3D uses four orthogonal views, which suits isolated objects but not occluded scenes. We keep each method's consistency machinery and replace only the choice of which views get edited.

Two other lines supply parts of the method. VcEdit unprojects diffusion cross-attention into 3D to find which Gaussians an edit concerns. We reuse that idea to set the weights $w_i$, but we do not run its iterative full-batch proxy loop. Both GaussianEditor papers (Fang et al., Chen et al.) show that restricting gradients to edit-region Gaussians preserves the rest of the scene and cuts fitting time. We use the same gradient masking when fitting from a few anchors, so unedited regions stay supervised by the original photographs.

On the selection side, Lazier Than Lazy Greedy supplies the stochastic greedy template. Our setting differs in that the ground set is continuous. That makes the per-step percentile argument, rather than a fixed finite pool, the relevant guarantee.

Robust Guarantees of Stochastic Greedy Algorithms constrains the design in two ways. First, stochastic greedy concentrates around the deterministic result under cardinality constraints but collapses under matroid constraints. Hard per-zone quotas ("two views from each side") would form exactly such a matroid, so we encode zone balance inside $\phi$ and the weights and keep the constraint a plain cardinality bound. Second, the same paper warns that concentration is weak at small $k$. Our $k$ is small, so seed-to-seed variance is something we measure, not something we assume away.

Research Program

The pipeline has four stages. In the first, edit-region weights come from unprojecting the diffusion model's text cross-attention on a handful of low-resolution renders into the Gaussians, as VcEdit does, and thresholding gives $R$.

In the second stage, candidate poses are sampled from a distribution over positions and look-at directions fitted to the captured cameras, widened to cover unobserved angles near the edit. Each candidate is scored by rasterizing a 128-pixel Gaussian-ID buffer, which gives visible Gaussians and their projected sizes in milliseconds. Greedy steps use the marginal gain of $F$.

In the third stage, the $k$ anchors (we sweep $k$ from 4 to 16) are rendered at full resolution and edited jointly with GaussCtrl's AttnAlign or DGE's spatio-temporal attention. Virtual anchors have no photograph, but both methods edit renders, so this is not a limitation.

In the fourth stage, the scene is fitted once. Edit-region Gaussians are supervised by the edited anchors, and all Gaussians are supervised by the original captured images outside the projected edit mask, with gradients masked as in GaussianEditor.

The analysis has two parts. The first tracks how edit quality scales with $k$ and with $F(S)$ itself. If $F$ does not predict quality across selection strategies, the objective is wrong. The second asks whether the benefit concentrates on localized edits in 360° scenes, where we expect it to, as opposed to whole-scene style edits, where uniform views should already cover everything.

Baselines

The naive reference point is the same four-stage pipeline with $k$ anchors drawn uniformly at random from captured cameras. A second naive point uses farthest-point sampling over camera centers, the standard engineering heuristic for spread-out views. Both run with identical $k$, diffusion settings, fitting iterations, and seeds. Beating them means higher CLIP directional similarity and lower cross-view warping error at the same $k$, with the gap growing for localized edits.

The strongest existing methods are full GaussCtrl and full DGE, run as published on the IN2N dataset and Mip-NeRF 360 scenes on the same GPU. DGE reports a CLIP similarity of 0.226 at roughly 4 minutes per edit on IN2N (DGE), and GaussCtrl reports about 9 minutes on an RTX A5000. Mip-NeRF 360 editing scores for these methods: no reported result found on Lacuna. We therefore rerun them ourselves. Beating them means matching DGE's CLIP similarity within noise at well under half its wall-clock time, or exceeding it at equal time. It also means better PSNR outside the edit region against the original scene.

The algorithmic competitor is deterministic lazy greedy on a fixed dense grid of a few thousand candidate poses, given the same selection-time budget as the randomized sampler. A cheap-compute competitor keeps all views but cuts diffusion steps and resolution until it matches our wall-clock time.

Three ablations isolate the key ingredients. The first replaces the relevance weights $w_i$ with uniform coverage of the whole scene. The second sets $\phi(c)=\min(c,1)$, which is plain coverage with no reward for overlap. The third restricts candidates to captured cameras only, which isolates the value of virtual poses. A fourth variant imposes hard per-zone quotas instead of the concave objective, to test whether the matroid fragility shows up in practice.

Why not just…?

Why not just use uniform or farthest-point views? For whole-scene stylization they may well be enough, and we expect a small gap there. For a small edit inside a cluttered 360° scene, spread-out views spend most of their pixels on irrelevant geometry and can miss occluded sides of the target entirely. DGE itself notes difficulty with occlusions and non-orbital trajectories. The weighted-versus-unweighted ablation and the split between localized and global edits will show directly whether edit-aware selection matters. If it does not, the proposal reduces to "edit fewer views," which is itself a useful result.

Why not just run deterministic greedy? Over the 50–300 captured cameras, deterministic lazy greedy is cheap, and we expect it to tie with the randomized version. This is the main threat to the randomized component, and we say so plainly. Randomization earns its place only once virtual poses are admitted, because exhaustively scoring a dense pose grid costs renders proportional to its size. On that point the stochastic greedy guarantee in Lazier Than Lazy Greedy does not save anything, since its total samples still scale with $n$. The fixed-sample percentile rule does. The captured-only ablation and the matched-time grid comparison settle this. If virtual anchors add nothing, we will report deterministic greedy over captured views as the recommended method.

Why not just use a feed-forward 3D editor? Shap-Editor edits in about one second inside a 3D generator's latent space, and Nano3D edits TRELLIS latents without training. These operate on generated, object-centric assets. Nano3D's own measurements show plain TRELLIS editing drifting in shape, with Chamfer distance 0.019 versus 0.013 for its grafted variant. Neither applies to a photographed unbounded scene. Our setting is real captures, where a feed-forward encoder for the whole scene does not exist in the file's literature.

Why not just cut diffusion steps on all views? This attacks the same latency without any selection. It is included as a matched-time baseline. If it matches our quality, selection is unnecessary.

Proposed Experiments

We will use IN2N scenes and the Mip-NeRF 360 scenes, with a prompt set split into localized edits (objects or parts) and global edits (style, season). Metrics are CLIP similarity and CLIP directional similarity, which are comparable to DGE's 0.226 and VcEdit's 0.2108 directional score on its own captures. We also measure cross-view consistency as photometric error after depth-based warping between held-out renders, PSNR and LPIPS outside the edit mask against the original scene, and end-to-end wall-clock time split into selection, diffusion, and fitting.

Three results would be decisive. The first is whether, on localized Mip-NeRF 360 edits at $k=8$, edit-aware stochastic selection beats farthest-point and random anchors on directional similarity and warping error across five seeds, with seed variance below the gap. The second is whether its selected sets reach $F(S)$ within a few percent of dense deterministic greedy at a fraction of the selection time. The third is whether the full pipeline matches DGE's quality in clearly less time. If $F(S)$ correlates with quality across all strategies, the objective is validated. If not, the coverage formulation is the wrong lens and we will say so.