Introduction
Autonomous driving (AD) systems rely heavily on high-fidelity simulations to test edge cases and rare scenarios safely. To create these virtual environments, researchers use Novel View Synthesis (NVS) techniques—such as Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS)—to reconstruct photorealistic 3D scenes from real-world driving logs. However, a critical gap exists in how these models are evaluated: while they excel at interpolating views along a recorded path, they often struggle when the virtual vehicle deviates from that path—a necessity for true closed-loop simulation.
The Challenge of Cross-Lane Synthesis
In a realistic simulation, a vehicle must be able to react, steer, and change lanes. This requires the NVS model to render "cross-lane" views—perspectives offset by several meters from the original training trajectory. Current benchmarks primarily evaluate interpolation between existing frames, masking the significant performance degradation that occurs when the camera moves laterally. Without ground-truth data for these offset views, it has been difficult to quantify how well current state-of-the-art models handle the geometric complexities of shifting perspectives.
The XLD Dataset
To address this, the Cross-Lane Dataset (XLD) was developed as a synthetic benchmark designed to stress-test NVS methods in off-trajectory scenarios.
XLD was generated using the CARLA simulator (based on Unreal Engine) to provide ground-truth data for parallel trajectories that are logistically difficult to capture in the real world.
- Composition: The dataset features 6 distinct urban and suburban scenes with varying weather conditions (sunshine, rain) and lighting.
- Data Streams: Each scene includes 150 waypoints with data captured from three RGB cameras (front, left-front, right-front) and a 64-beam LiDAR.
- Evaluation Protocol: Models are trained on a primary trajectory and tested against ground-truth images captured at parallel offsets of 0m, 1m, 2m, and 4m.
Benchmarking Methodology and Insights
The authors evaluated a suite of leading NeRF-based methods (e.g., EmerNeRF, UC-NeRF, MARS) and 3DGS-based methods (e.g., PVG, DC-Gaussian, 3DGS). By training these models on the primary trajectory and evaluating them on the offset paths using standard metrics like PSNR, SSIM, and LPIPS, several key insights emerged:
- NeRF Robustness vs. 3DGS Precision: While 3DGS methods like PVG achieve superior results on standard interpolation (up to 37.78 dB PSNR at 0m offset), they tend to overfit the training path. NeRF-based methods, particularly EmerNeRF and UC-NeRF, proved significantly more robust at large offsets, outperforming 3DGS models by up to 3.43 dB PSNR at a 4m offset.
- The Geometry Bottleneck: Accurate 3D geometry is the primary bottleneck for cross-lane synthesis. Models that rely too heavily on photometric loss without strong geometric priors often fail to maintain correct spatial relationships, such as proper occlusion of background elements, when the viewpoint shifts.
- Scene Decomposition: Explicitly separating the scene into static backgrounds, dynamic objects, and the sky allows models to handle viewpoint changes more effectively than unified volume approaches.
- Multi-Camera Advantage: Training with a wider baseline (using side cameras) drastically improves performance. For instance, EmerNeRF saw a 1.82 dB PSNR boost at 4m offsets when trained with three cameras instead of one, as the additional views provide better coverage of the scene's geometry.
Conclusion
The XLD benchmark highlights a significant limitation in current NVS models: their inability to generalize to off-trajectory views. While synthetic, XLD provides a rigorous framework for developing the next generation of robust NVS algorithms necessary for truly interactive, closed-loop autonomous driving simulations. Future work will focus on bridging the sim-to-real gap by curating real-world datasets with genuine cross-lane ground truth.
Summary updated