Driving simulators and robot teleoperation need a virtual camera that can leave the path the real camera took: swerve a lane over, look sideways, and do it at interactive frame rates while cars and pedestrians keep moving. Current methods offer two incompatible halves of this. 3D Gaussian Splatting (3DGS) represents a scene as millions of small colored ellipsoids that a GPU can draw at around 100 frames per second, but it overfits the recorded camera path. Neural radiance fields (NeRFs) store the scene as a neural network queried hundreds of times per pixel. They are far slower to render, yet on a controlled cross-lane benchmark they hold up better when the camera moves sideways. We propose to train the fast representation on what the slow one predicts at camera poses nobody recorded. The slow model is used only during training, so it adds nothing to rendering time.
The specific bet is this. The off-path robustness of decomposed neural fields such as EmerNeRF can be transferred into a dynamic Gaussian scene by supervising the Gaussians with the field's renders at synthetic laterally shifted poses, with each pixel weighted by how confident the field is there. If the bet holds, one model would match the in-path quality of the best Gaussian methods, come close to the field's quality several meters off-path, and render in real time. No method in the literature we surveyed occupies that combination.
Core Claim
The XLD benchmark (XLD) renders CARLA driving scenes from trajectories shifted 0, 1, 2 and 4 meters sideways from the recorded one. It reports a striking reversal. The Gaussian method PVG reaches 37.78 dB PSNR (peak signal-to-noise ratio) on the recorded path. At a 4 m offset, however, NeRFs that separate static background, moving objects and sky (EmerNeRF, UC-NeRF) beat the 3DGS models by up to 3.43 dB. XLD calls the 3DGS failure "dashcam bias." VEGS traces part of it to Gaussians stretching into needles along the training rays, which it calls "lazy covariance." XLD is a benchmark only: it proposes no fix and measures no latency. On the other side, EmerNeRF learns a static field, a dynamic field and a 3D motion field without labels, but it renders by marching rays through a network and cannot run in real time.
Our claim is that the gap reflects what each representation has learned about unseen poses, not how fast it can be evaluated. A field with smooth, decomposed geometry generates plausible off-path views. A Gaussian scene trained on those views, in addition to the real ones, should inherit most of that plausibility and still render at Gaussian speed. Concretely, the student is trained with
$$ \mathcal{L} = \sum_{v \in V_{\text{real}}} \ell\big(I^{S}_v, I_v\big) + \lambda \sum_{p \in P_{\text{off}}} \sum_{x} w_p(x)\, \ell\big(I^{S}_p(x), I^{T}_p(x)\big), $$
where $V_{\text{real}}$ are the captured views with images $I_v$, and $P_{\text{off}}$ are sampled poses shifted laterally and in yaw at the same timestamps. $I^{S}$ and $I^{T}$ are the student and teacher renders, $\ell$ is the usual L1 plus structural-similarity photometric loss, $\lambda$ balances the two terms, and $w_p(x)\in[0,1]$ is a per-pixel confidence for the teacher. The confidence is built from the teacher's accumulated opacity together with the disagreement between two teacher seeds. Pixels where the teacher is guessing, such as never-observed disocclusions, receive low weight instead of being imposed on the student.
Research Program
The work has three parts. The first is diagnostic, because it determines whether distillation is even the right lever. XLD attributes NeRF's advantage loosely to decomposition. We will separate the contributing factors by giving a dynamic Gaussian model each ingredient EmerNeRF has, one at a time: an explicit sky model, a static/dynamic split, and depth supervision. If decomposition alone closes most of the 3.43 dB, the right proposal is "decompose your Gaussians," and we will report that and stop there. Distillation is justified only by the residual that remains.
The second part is the distillation itself. The teacher is EmerNeRF, trained per scene on the recorded cameras. The student is a decomposed dynamic Gaussian scene. Static Gaussians are initialized from LiDAR, and dynamic Gaussians are moved by a small deformation network in the style of GauFRe, which renders at 96 FPS. The sky is a separate environment term. Because the teacher has its own dynamic field and motion field, its off-path renders at time $t$ already contain moving cars at the right place, so moving objects are supervised from the new viewpoints too. VEGS does not get this: it isolates cars with an external 3D box detector, and its generative prior only addresses static texture. Off-path poses are sampled with a curriculum, starting at small offsets and widening to about 4 m and ±30° yaw, and pseudo-views are refreshed as the student improves. We will also try one cheap extension: distilling teacher depth alongside teacher color, to test whether geometry or appearance carries the transfer.
The third part handles regions no teacher can know, namely areas hidden in every frame. Here the confidence weights fall to near zero and the student falls back to its own regularization, which follows the VEGS normal prior that flattens Gaussians into disks. We will measure these regions separately rather than let them blur the headline number. We deliberately do not put a diffusion model in the rendering loop. Methods that do, such as ViewExtrapolator and Vid-CamEdit, run iterative denoising over many frames and are far from real time. Using a generative prior during training only, as VEGS does, remains compatible with our design and is tested below as an add-on.
Baselines
The naive reference point is a dynamic Gaussian model trained only on the recorded cameras. We use PVG, the best in-path model on XLD at 37.78 dB at 0 m, together with a GauFRe-style deformable model under our exact initialization. This shows what dashcam bias costs without intervention. To beat it, our model must win clearly at 2 m and 4 m while losing no more than a few tenths of a dB at 0 m.
The strongest off-path quality competitor is EmerNeRF itself, run with the authors' settings on the same XLD scenes and cameras. XLD reports it beating 3DGS by up to 3.43 dB at 4 m and gaining 1.82 dB at 4 m when trained on three cameras instead of one. The file gives these only as gaps, not as absolute 4 m PSNR values, so we will rerun it ourselves. It is the teacher, so the student cannot be expected to exceed it off-path. Success means coming within about 0.5 dB of EmerNeRF at 4 m while rendering at 60 FPS or more, and matching or beating PVG at 0 m. That combination is the Pareto point no existing model occupies.
The strongest real-time extrapolation competitor is VEGS. It combines LiDAR initialization, monocular normal priors, box-based car separation and Stable Diffusion score distillation, and renders at about 108 FPS. On KITTI-360 under a ±60° yaw protocol it reports FID 124.4 and KID 0.0561. These are distribution-distance metrics; lower is better. We will run VEGS on XLD with its released recipe, and compare against it on KITTI-360 under its own protocol. There we must match or improve FID and KID at comparable frame rate, and do especially well on moving cars, where VEGS depends on box detections.
Two ablations isolate the key ingredient. The first is "decomposed Gaussians, no teacher," from the diagnostic stage. The second is "teacher without confidence weights," which shows whether filtering where the teacher is guessing matters. Every Gaussian variant gets the same initialization, the same number of optimization steps, and the same hyperparameter search budget. We report teacher training time separately and in full, so that the extra offline cost is visible rather than hidden.
Why not just…?
Why not just add VEGS-style LiDAR and normal priors to a dynamic Gaussian model? This is the most serious threat. It might close much of the gap without any teacher. The normal prior flattens the needle-like Gaussians, and LiDAR fixes depth along the path. What these priors cannot provide is appearance at unseen angles or moving objects viewed from new positions. VEGS has to add diffusion-based score distillation for texture and still separates cars with external boxes. Our ablation grid includes "priors only," "teacher only" and "both," evaluated with XLD ground truth at every offset. If "priors only" comes within noise of "both," we will conclude that the teacher is unnecessary.
Why not just clean up off-path renders with a video diffusion model? ViewExtrapolator improves LPIPS on LLFF-Extra from 0.429 for plain 3DGS to 0.378. (LPIPS is a learned perceptual distance; lower is better.) But it denoises iteratively per sequence, which breaks the real-time requirement. Its corrections also never flow back into the 3D scene, and it struggles with fast-moving objects. We will report it on XLD as a quality reference with its measured frame rate, so the trade-off is shown rather than asserted.
Why not just bake EmerNeRF into a fast grid? SNeRG (Baking Neural Radiance Fields for Real-Time View Synthesis) achieves real-time rendering by precomputing a sparse voxel atlas. That atlas assumes static geometry, so a dynamic scene would need one grid per timestep, with memory growing in proportion to the capture length. Distillation gives a compact, deformable scene instead. We will report student memory, keeping in mind that MEGA shows dynamic Gaussians can be compressed heavily, up to 190× storage reduction at 83.14 FPS, if memory becomes the bottleneck.
Why not just record with more cameras? XLD's 1.82 dB multi-camera gain for EmerNeRF shows that real side views help. They are not always available, though: fleet logs are often front-camera-heavy, and KITTI-360 is forward-facing. We will train every method on both the single-camera and the three-camera XLD settings. If the teacher's benefit vanishes when three cameras are available, that is a useful bound on where the method matters.
Proposed Experiments
The primary testbed is XLD, because it is the only benchmark in our material with ground-truth images at controlled lateral offsets. We report PSNR, SSIM and LPIPS at 0, 1, 2 and 4 m, frames per second at the evaluation resolution on one fixed GPU, and training time split into teacher and student. Using teacher opacity across all training frames, we also compute a mask of regions never observed. Metrics are reported inside and outside that mask, which separates "transferred knowledge" from "unknowable content."
Real-world transfer is tested on KITTI-360 under the VEGS extrapolation protocol (FID, KID, FPS). On the NOTR dynamic sequences from Waymo we run an in-path sanity check, confirming that pseudo-view supervision does not degrade ordinary reconstruction of moving objects.
Three outcomes decide whether the proposal holds. First, on XLD at 4 m in the single-camera setting, the confidence-weighted student must recover at least half of the PVG-to-EmerNeRF gap beyond what decomposition plus VEGS priors already recover. Second, it must do so at 60 FPS or more and within 0.3 dB of PVG at 0 m. Third, removing the confidence weights must hurt specifically inside the never-observed mask, showing the weights do what they are designed to do. If the first check fails, the result is still informative: the off-path advantage of decomposed fields comes from decomposition and priors that Gaussians can adopt directly, and the field is not needed as a teacher.
References
- XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis
- VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting Using Learned Priors
- EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
- GauFRe: Gaussian Deformation Fields for Real-Time Dynamic Novel View Synthesis
- Novel View Extrapolation with Video Diffusion Priors
- Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry
- Baking Neural Radiance Fields for Real-Time View Synthesis
- MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes