Scene reconstructions such as Neural Radiance Fields (NeRFs, networks that learn a 3D scene from a few photos and can render it from new camera positions) are now a common source of extra training images for spatial perception models, such as networks that predict a camera's position from a single image. The standard recipe works: render thousands of new viewpoints offline, discard the ugly ones, and train on the rest. It is also wasteful and blind. One system renders 100,000 views and throws away 90,000 using generic image-quality scores. Another finds that the same recipe gives no benefit, or even hurts, on indoor depth benchmarks, where close-up surfaces expose reconstruction artifacts. The filter asks "does this image look good?" when the question that matters is "will training on this image make the model better on real photos?"
We propose to answer that second question directly and cheaply, during training. 3D Gaussian Splatting (a scene representation built from millions of small colored blobs that renders at hundreds of frames per second) makes it affordable to render candidate views on the fly. For each candidate we then ask whether the weight update it would cause points in the same direction as the update from a small set of held-out real frames. Views that agree are kept. Views that are informative but broken, such as floaters, blur, or holes in extrapolated regions, produce large losses but gradients that point away from real data, so they are rejected. In short: fast rendering lets the training loop propose viewpoints, and a handful of real images decide which proposals become training data.
Core Claim
For camera pose regression and monocular depth estimation, choosing synthetic views by how well their gradients align with gradients from real validation frames will match or beat offline render-and-filter pipelines. It will do so with an order of magnitude fewer rendered views, and it will remove the negative transfer seen when rendering artifacts dominate. The key difference from hard-example mining is the real-data anchor. Mining that looks only for high loss preferentially selects exactly the corrupted renders that open the synthetic-to-real gap.
Write $\theta$ for the parameters of the downstream model, $v$ for a candidate camera pose rendered into image $x_v$ with its rendered label $y_v$, and $R$ for a small set of real frames with ground truth. The selection score is
$$ s(v) = \cos\big(\nabla_\theta \ell(x_v, y_v; \theta),\ \nabla_\theta \tfrac{1}{|R|}\textstyle\sum_{r \in R} \ell(x_r, y_r; \theta)\big), $$
where $\ell$ is the task loss. The score is a first-order estimate of how much one step on $v$ lowers real loss. We will compute gradients only for the final layers to keep the cost per candidate small.
Relation to Prior Work
Three papers establish that radiance-field augmentation helps spatial tasks, and each motivates this proposal through the gap it leaves. LENS trains a NeRF variant on unconstrained photo collections, renders views on a regular 3D grid inside navigable space, and cuts pose-regressor error by about 60%. Its authors note that rendering takes days and that rendering artifacts inject noise into the regressor. Augmenting with NeRFs perturbs real poses with a fast NeRF implementation, keeps the best 10% of renders using referenceless quality metrics (BRISQUE, ClipIQA), and doubles a lightweight regressor's accuracy. NeRFmentation renders shifted trajectories for driving depth estimation and filters whole scenes with a fixed quality threshold. It helps on KITTI and Waymo but not on NYU-Depth V2 or Replica. All three decide which views to use without consulting the downstream model or any real image. That decision is what we replace.
Two papers supply the idea of optimizing data against real validation loss, and both constrain our design. Neural-Sim differentiates through a NeRF renderer to tune camera, zoom and lighting distributions against detection loss on real images, with 20–60% gains. It needs memory-saving tricks, is limited by rendering latency, and optimizes a few global distribution parameters rather than choosing individual views. We avoid differentiating through the renderer altogether and score each view individually. RLS3 trains a reinforcement-learning agent in Unity to seek configurations where a vision-language model fails, and reports about 40% better sample efficiency than random sampling. Its signal is failure alone. In a radiance field, failure is often caused by the renderer rather than the task, and that is precisely the case our real-anchored score is designed to catch.
The rendering side supplies the method. 3D Gaussian Splatting renders in real time, and Cut-and-Splat and MapGS already use it to generate data. Both do so offline, and neither lets training feedback steer generation. GS-CPR uses splatting renders plus an exposure-matching color transform to refine poses at test time. We reuse its color transform as an optional photometric correction. Unlike GS-CPR, which refines a pose already estimated, we use rendering to train a regressor that needs no initial pose. The sensor-noise finding in Adversarially Masking Synthetic to Mimic Real, where clean synthetic scans invite shortcuts, predicts that pristine-looking renders are not automatically good training data. Our score tests that prediction directly.
Research Program
We fit one Gaussian Splatting model per scene from the standard training frames and hold out a small real anchor set $R$ drawn from those same training frames, never from the test split. During regressor training, each step proposes $K$ candidate poses. Half are Gaussian perturbations of real training poses and half are sampled uniformly from free space estimated from splat density, following the free-space idea in LENS. Each candidate is rendered to RGB (and to depth for the depth task). We compute last-layer gradients, score every candidate with $s(v)$, and train on the top $B$ together with a real minibatch. The anchor gradient is refreshed every few hundred steps. To stop the sampler collapsing onto views that closely resemble $R$, we keep a small fraction of random views and cap how close selected poses may lie to one another.
The analysis has three parts. First, we will record where selected and rejected views lie in pose space, measured as distance from the nearest training camera and as rendering-uncertainty proxies. This tells us whether the score is really learning to avoid extrapolation artifacts or only reproducing a distance threshold. Second, we will track the fraction of candidates accepted over training to see how the useful synthetic distribution shifts as the model improves. Third, we will measure total renders and GPU hours against the offline pipelines.
Baselines
The naive reference is the same regressor trained only on real training frames plus the anchor frames $R$. Folding $R$ back into training matters: it ensures any gain comes from how the real frames are used, not from having more of them.
The strongest existing method for pose regression is offline radiance-field densification in the style of LENS and Augmenting with NeRFs. We will reimplement it on the same Gaussian Splatting model: grid and perturbation sampling, a quality filter that keeps the top 10%, and training on the result together with real frames plus $R$. On 7Scenes, LENS reports average translation error falling from 0.22 m to 0.08 m, and on Cambridge Landmarks from 0.92 m to 0.39 m (LENS). Augmenting with NeRFs reports at least a 2x gain over its unaugmented baseline, but the file contains no per-scene numbers. Because regressors differ across papers, we treat these figures as context and compare against our own reimplementation. Every method gets the same regressor, the same number of gradient steps, the same count of synthetic images seen, and the same hyperparameter search budget. Beating this baseline means lower median translation and rotation error with at least 5x fewer renders, and no scene where our method is worse by more than seed noise.
For depth, the reference is NeRFmentation-style trajectory augmentation. It reports a Waymo out-of-distribution relative error of 0.220, against 0.297 for Fourier Domain Adaptation and 0.287 for style transfer (NeRFmentation). On the indoor sets where that method failed to help, the bar is simpler: improvement over real-only training.
Three ablations isolate the key ingredient, the real-gradient anchor. The first selects the highest-loss candidates, a model-guided but unanchored scheme in the spirit of RLS3. The second selects by distance from training cameras only, a purely geometric trust region. The third selects at random from the same online proposals.
Why not just…?
Why not just render a large offline pool with Gaussian Splatting and filter it, now that rendering is cheap? This is the most serious alternative, because splatting removes the multi-day latency that made LENS expensive. If online proposal adds nothing, the contribution shrinks to the scoring rule. We therefore include a variant that applies our score to a fixed pre-rendered pool of 10k, 100k or 1M views. If it matches the online version, we will report that the score is what matters and that online generation only saves storage. We expect the online version to win where pose space is large, as in Cambridge and the far-away Hangar and Dronerace scenes from Augmenting with NeRFs, because a fixed pool thins out exactly where the model needs new views.
Why not just mine the hardest synthetic views? Loss-driven mining treats high loss as informative. In a radiance field, the highest-loss renders are often floaters and holes in unobserved regions, which is the failure mode catalogued for extrapolated NeRF views. The loss-only ablation tests this directly. We predict it will select views far from training cameras and will reproduce NeRFmentation's indoor degradation, while the anchored score will not.
Why not just fine-tune on the real anchor frames? The naive baseline already gives every method these frames as training data. If real-only training with $R$ matches our method, synthetic data is not helping and we will say so. A sweep over the size of $R$ (1%, 5%, 10% of training frames) shows how few real frames the anchor needs to be useful.
Why not just refine poses at test time with GS-CPR? GS-CPR reaches 93.1% recall at 2 cm/2° on 7Scenes, improving DFNet's median error from 6 cm/1.93° to 1.1 cm/0.34° (GS-CPR). No trained regressor is likely to match that. However, it adds up to about 180 ms per query, requires the scene model and a matching network at deployment, and fails when the initial pose is off by more than about 8 m or 50°. Our goal is a better initializer that runs in a few milliseconds. We will report GS-CPR as an upper reference, and also report GS-CPR applied on top of our regressor, to show whether a better initializer reduces its failures.
Proposed Experiments
Pose regression runs on 7Scenes and Cambridge Landmarks with a lightweight regressor of the kind used in Augmenting with NeRFs, plus Hangar and Dronerace for far-away queries. We report median translation and rotation error, recall at 5 cm/5°, renders consumed, and GPU hours, averaged over three seeds.
Depth estimation runs on KITTI with Waymo as the out-of-distribution test, and on NYU-Depth V2 and Replica. We report relative error and RMS error.
Two outcomes would decide the question. The first is the indoor depth result. If anchored selection turns NeRFmentation's null or negative indoor effect into a gain while loss-only mining does not, that is direct evidence that task-anchored selection closes a synthetic-to-real gap that quality filters miss. The second is the render budget at equal accuracy against the offline top-10% pipeline, along with the fixed-pool comparison, which shows whether online proposal matters beyond the score itself.
A negative result would also be informative. If distance-only selection matches the anchored score, the useful signal is purely geometric, and a much simpler trust-region rule should replace ours.
References
- LENS: Localization enhanced by NeRF synthesis
- Augmenting with NeRFs: Fast Relocalization on Densified Datasets
- NeRFmentation: Improving Monocular Depth Estimation with NeRF-Based Data Augmentation
- Neural-Sim: Learning to Generate Training Data with NeRF
- RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering
- Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation
- MapGS: Generalizable Pretraining and Data Augmentation for Online Mapping via Novel View Synthesis
- GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation