Fast 3D reconstruction models now turn a few photos into a renderable 3D scene in under a second. They do this by predicting thousands of small colored blobs, called 3D Gaussians, in one or a few network passes, instead of fitting them to each scene over thousands of gradient steps. These models are trained only to make rendered images match photographs. As a result, they happily place "floaters," semi-transparent blobs hanging in empty air. A floater can explain the input photos well and still block a hallway the camera physically walked through. Every existing method that enforces such physical or geometric rules on Gaussian scenes goes back to slow per-scene optimization. Planar Gaussian Splatting spends about 16 minutes per ScanNet scene, and PhyCAGE nests a physics simulator inside an iterative distillation loop. The speed gained by feed-forward models is lost the moment we ask the scene to be physically valid.
We propose to borrow the central trick of dual-population evolutionary search, which splits candidates into a feasible pool (optimized for quality) and an infeasible pool (optimized only to stop breaking rules), and build it into a learned recurrent refiner. The main scene hypothesis is refined for image quality. A few lightweight "scout" hypotheses, restricted to the regions that break constraints, are refined only to reduce violations. A scout is merged back into the main hypothesis once it becomes valid. The bet is specific: when image fidelity and validity pull in opposite directions, as they do for a floater that explains the input views, a single refinement trajectory with a penalty term trades one for the other. Decoupled scouts can find valid configurations the main trajectory never reaches, at the cost of a few extra recurrent steps rather than minutes of optimization.
Core Claim
In a recurrent feed-forward Gaussian reconstructor, keeping a separate constraint-only update pool that crosses into the main hypothesis on feasibility will strictly dominate a single constraint-aware trajectory on the trade-off between rendering quality and constraint violation. The gain should be concentrated on regions where photometric and constraint signals conflict. It should also cost less than 2x in latency, keeping reconstruction within roughly the same order of magnitude as current recurrent models and far below per-scene optimization.
The claim is narrow on purpose. If a single-trajectory refiner that simply sees a constraint-violation map as an extra input matches our results, the dual-pool idea is unnecessary. The experiments are designed so that this outcome is visible.
Research Program
Where each ingredient comes from. The method substrate is ReSplat: Learning Recurrent Gaussian Splats. ReSplat initializes a compact set of about 246K Gaussians and runs four gradient-free recurrent steps. Each step renders the current scene, computes image-space error, routes that error back to the Gaussians with attention, and predicts residual updates. It reportedly matches 4,000 iterations of per-scene fitting at about 100x the speed. Its feedback is purely photometric, and it follows one hypothesis.
The dual-population logic comes from three sources. The first is the feasible–infeasible two-population genetic algorithm in Mario Level Generation From Mechanics Using Scene Stitching, which reports 87–100% playability against about 25–28% for greedy single-criterion search. The second is the push-then-pull dynamic of Push and Pull Search Embedded in an M2M Framework. The third is the decoupled constraint loop and objective loop of Bayesian optimization with active learning of design constraints using an entropy-based approach. These papers motivate the design: they show that decoupling beats penalized single-pool search when feasible regions are fragmented. They operate on tile grids and 5-dimensional alloy spaces, not on hundreds of thousands of continuous primitives. What we add is putting the pools inside a learned, unrolled refiner, so the search happens in a few amortized network steps rather than over many generations.
Constraints that need no ground truth at inference. We use two constraints that can be computed from input poses and the predicted Gaussians alone.
The first is trajectory navigability. The cameras in walkthrough datasets physically occupied their positions, so a tube of fixed radius around the interpolated camera path must be free of opaque Gaussians. This constraint behaves like a step function: a single blob can block the path.
The second is ray free-space. For pixels where the multi-view depth branch is confident, the space along the ray well in front of the predicted surface should carry little accumulated opacity.
We write the violation of scene $\mathcal{G}$ as
$$ V(\mathcal{G}) = \sum_{x \in \mathcal{T}} \max\big(0,\ \sigma_{\mathcal{G}}(x) - \tau\big) + \sum_{r \in \mathcal{R}} \max\big(0,\ A_{\mathcal{G}}(r) - \tau\big), $$
where the terms are defined as follows. $\mathcal{T}$ is a set of sample points in the camera-path tube. $\sigma_{\mathcal{G}}(x)$ is the opacity-weighted Gaussian density at point $x$. $\mathcal{R}$ is the set of confident free-space ray segments. $A_{\mathcal{G}}(r)$ is the opacity accumulated along segment $r$. $\tau$ is a small tolerance.
A region counts as feasible when its contribution to $V$ is zero.
Architecture. At each recurrent step, the main hypothesis receives ReSplat's photometric error features and is updated as usual. In parallel, we identify the Gaussians that contribute to $V$; this is typically a small fraction of the scene. For those Gaussians we spawn $K$ scouts, with $K$ between 2 and 4.
Each scout is a sparse set of residual latents attached only to the violating Gaussians. This resolves the memory tension noted in the file: we never duplicate the full primitive set, only a small subset. The scout update network is trained with only the violation objective. Its action space includes moving a Gaussian along its own input-view ray, which leaves input renders nearly unchanged; shrinking it; and fading its opacity.
The crossing rule works as follows. When a scout's region reaches zero violation, the scout replaces the main hypothesis's corresponding Gaussians. This happens only if the input-view photometric error in the affected pixels rises by less than a threshold $\epsilon$; otherwise the scout persists for another step. This is the "pull" stage; the constraint-only scout update is the "push."
Training. We unroll $T$ steps (4 to 6) and train end-to-end on RealEstate10K and DL3DV. The main pool is trained with the standard rendering loss on held-out views. The scout pool is trained with $V$ plus a weak held-out-view rendering loss on the merged result. Because $V$ and the crossing decision are non-differentiable at the boundary, we have two options for training the crossing gate: a straight-through estimator, or a simple REINFORCE-style signal whose reward is the drop in $V$ after merging. We will report which one is stable.
A practical note on code. The ReSplat codebase listed in the file, yh-yoon/ReSplat, is a different, degradation-oriented diffusion method that shares the name. The recurrent refiner will need to be reimplemented from its paper, or obtained from its authors, before any comparison is fair.
Baselines
The naive reference point is DepthSplat, a single-pass model that fuses a stereo cost-volume branch with a monocular depth prior. It is run unchanged and reconstructs a 12-view high-resolution scene in about 0.6 seconds. It anchors "fast but unconstrained": we expect its violation score to be highest. We will also report a trivial post-hoc filter on top of it, which deletes every Gaussian that contributes to $V$.
The strongest existing fast method is recurrent ReSplat. It is reported to beat DepthSplat by 3.5 dB PSNR on DL3DV 8-view synthesis and to be about 100x faster than per-scene 3DGS. The file gives no absolute PSNR for it. For absolute context, IDESplat reports 27.80 dB PSNR on RealEstate10K. For any constraint-violation metric on these methods, no reported result was found on Lacuna, so we will measure violation for all baselines ourselves. ReSplat, our method and all ablations use the same backbone, the same training data, the same number of training iterations, and equal hyperparameter-search budgets. Recurrent step counts are matched by total compute, not by step index. This means ReSplat is also run at $T$ steps chosen so its latency equals ours, which removes the "you just ran more steps" objection.
The key ablation is ReSplat with the violation map concatenated to its error features and $\lambda V$ added to its training loss, swept over $\lambda$. This is the single-pool, constraint-aware version of our method, and it isolates the pool split as the only difference. Beating it means our quality–violation Pareto front lies strictly outside the entire $\lambda$-sweep front on held-out ScanNet and DL3DV scenes. Concretely, at matched novel-view PSNR (within 0.2 dB), our violation must be meaningfully lower, which we will judge with scene-level bootstrap confidence intervals. The second ablation keeps the scouts but removes the crossing gate and simply averages scout and main updates; it tests whether the push-then-pull structure matters.
The quality ceiling is per-scene 3DGS with the same $V$ added as a penalty, run to convergence. We also report Planar Gaussian Splatting's reported ~16 minutes per ScanNet scene as a runtime reference for constrained per-scene fitting. We do not expect to beat this ceiling on quality. The target is to close most of the violation gap at a small fraction of its runtime.
Why not just…?
Why not just give ReSplat the violation map as another feedback channel? This is the real threat. It is also why it serves as the key ablation rather than a strawman. The evolutionary evidence argues against it: penalized single-pool search stalled on fragmented feasible sets in the Mario comparison and in the push-pull work. But that evidence comes from discrete and low-dimensional problems and may not transfer to a learned refiner. The experiment settles the question directly. We stratify test regions by a conflict score, the increase in input-view error caused when the violating Gaussians are removed. If our method's advantage does not concentrate in the high-conflict stratum, or vanishes entirely, we conclude the split is unnecessary and report that.
Why not just add a penalty to the training loss of a single-pass model? A penalty term trained into DepthSplat changes the average prediction but gives no chance to correct a specific scene at inference. Planar Gaussian Splatting shows that even per-scene penalty balancing needs careful tuning and remains slow. We include DepthSplat trained with $\lambda V$ as an additional row. We expect it to lose to both recurrent variants, because it has no test-time feedback at all.
Why not just delete or project the offending Gaussians afterward? Floaters often carry real appearance. Deleting them opens holes, and moving them breaks their alignment with the pixels they came from. PhyCAGE found that enforcing non-penetration required integrated simulation with appearance frozen and re-optimized, not simple projection. Our scouts include "slide along the input ray," which is a smarter projection. The post-hoc filter row measures how much a dumb projection loses in held-out-view PSNR. If it loses little, the problem is easier than we think, and the contribution shrinks to a training recipe.
Why not just sample several hypotheses and keep the most valid one? Best-of-$K$ selection over whole scenes spends $K$ times the memory. It also cannot combine the valid region of one sample with the good appearance of another. Scouts are local and mergeable. We will run best-of-$K$ with $K$ matched to our scout count, using stochastic initial depths, to check this.
Proposed Experiments
We train on RealEstate10K and DL3DV. Evaluation happens in two places. The first is DL3DV 8-view synthesis for photometric quality (PSNR, SSIM, LPIPS). The second is ScanNet and ScanNet++, whose ground-truth meshes let us audit validity independently of $V$: we measure the opacity mass inside ground-truth free space, and the fraction of camera-path segments blocked in a floor occupancy map built from the Gaussians against one built from the mesh. For every method we report latency and peak memory on a single GPU.
The first decisive check is the Pareto comparison against the constraint-aware single-pool ablation at matched latency, as specified in the baselines. The second is the conflict-stratified breakdown. Our prediction is that almost all of the gain sits in the top conflict tertile and near-zero gain appears in the bottom tertile. Uniform gain would suggest the benefit comes from extra capacity rather than decoupling. The third is population imbalance, the failure mode flagged for dual-population methods. We log how often scouts never cross, and how often they cross immediately, which would make them pointless. We then vary $K$ and $\epsilon$ to find whether a stable regime exists.
The fourth check is cross-dataset transfer to DTU and ACID. There we expect violation reduction to carry over, because both constraints come only from poses and depth confidence, while the photometric gap should track ReSplat's.
The project succeeds if the dual-pool refiner reaches most of the violation reduction of penalized per-scene fitting in under a few seconds, and does so in a way the single-trajectory ablation provably cannot match.
References
- ReSplat: Learning Recurrent Gaussian Splats
- Mario Level Generation From Mechanics Using Scene Stitching
- Push and pull search embedded in an M2M framework for solving constrained multi-objective optimization problems
- Bayesian optimization with active learning of design constraints using an entropy-based approach
- DepthSplat: Connecting Gaussian Splatting and Depth
- IDESplat: Iterative Depth Probability Estimation for Generalizable 3D Gaussian Splatting
- Planar Gaussian Splatting