TL;DR
DIRT (Depth-completed Image-based Rendering with Two-band blending) is a non-neural, optimization-free pipeline that enables high-quality novel-view synthesis in milliseconds. By combining hierarchical sparse depth completion with frequency-aware blending, it allows for real-time scene exploration during capture across diverse modalities, including LiDAR, thermal, and handheld RGB-D sensors.
What Problem Does This Paper Address?
State-of-the-art novel-view synthesis (NVS) methods, such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS), typically require lengthy per-scene optimization—often taking minutes or hours—before they can render a scene from a new perspective. This makes them unsuitable for "live" applications like autonomous driving previews or interactive handheld scanning, where the delay between capture and rendering must be minimal.
While some "generalizable" neural networks attempt to skip optimization, they often struggle to generalize to new environments (e.g., switching from daylight to night-time or from RGB to thermal cameras). The authors address the need for a low-latency, sensor-agnostic NVS approach that doesn't rely on training data or scene-specific "baking."
Methods
DIRT returns to the roots of Image-Based Rendering (IBR) but modernizes it with a robust, non-neural pipeline. The process consists of three main stages:
- Geometric Proxy Construction: The system takes sparse point clouds (from LiDAR or RGB-D sensors) and projects them into the target novel view. To fill the resulting "holes," the authors use a hierarchical filling strategy. This recursive, image-space algorithm densifies the depth map while remaining anchored to actual measurements rather than "hallucinating" geometry like a neural network might.
- Warping and Disocclusion: Posed auxiliary images are warped onto this dense geometric proxy. To handle areas that should be occluded, the system renders the proxy as a triangle mesh from the perspective of the auxiliary cameras, performing a depth test to identify and mask out hidden regions.
- Two-Band Blending: To avoid the "ghosting" artifacts common when blending multiple images, DIRT employs a frequency-aware strategy. It decomposes images into high and low-frequency bands using a Laplacian Pyramid. Low frequencies (colors/lighting) are blended smoothly over a wide area, while high frequencies (sharp details) are selected from the most reliable source view. Blending weights are determined by a softEDT (approximate distance transform) to minimize seams near occlusion edges.
Datasets
The authors evaluated DIRT across a wide variety of challenging scenarios:
- Automotive: nuScenes, Waymo, and Argoverse 2 (LiDAR + multi-camera setups).
- Handheld RGB-D: The LiveNVS dataset.
- Conventional NVS Benchmarks: MipNeRF 360 and Tanks and Temples (for offline comparison).
- Cross-Modal: RGBT-Scenes (Thermal imagery) and night-time driving sequences.
Key Insights and Ideas
- Non-Neural is a Feature, Not a Bug: By avoiding neural components, DIRT is inherently robust to "domain shift." It works on thermal data or low-light night scenes without needing to be retrained, whereas neural baselines often fail to converge or produce blurry results in these settings.
- Measurement-Anchored Geometry: Unlike learned depth completion that might guess geometry, DIRT’s hierarchical filling stays faithful to the sensor data. This is critical for "capture guidance," where a user needs to see exactly what has (and hasn't) been recorded.
- Frequency-Aware Blending: The "aha moment" for visual quality is the two-band blending. It allows the system to hide small misalignments in the geometry by smoothly transitioning the "glow" of the scene while keeping the high-frequency textures sharp.
Key Results
- Speed: DIRT achieves "capture-to-image" latencies of 69ms to 135ms, significantly faster than optimization-based methods that take seconds or minutes.
- Perceptual Quality: In live reconstruction scenarios, DIRT consistently outperformed prior methods (like On-the-fly NVS and Long-LRM) in LPIPS, a metric that tracks human-perceived image quality.
- Robustness: In night-time and thermal tests, DIRT maintained high fidelity while competing neural methods often failed to produce a consistent scene representation. For handheld RGB-D captures, DIRT improved PSNR by up to 11% over the next best state-of-the-art live method.
Limitations and Future Work
- Edge Artifacts: While two-band blending helps, DIRT can still exhibit seam-like artifacts at depth discontinuities if the reconstructed geometry is slightly misaligned with the images.
- Static Scene Assumption: The current pipeline assumes the world is static. While the point accumulation strategy filters out some transient objects (like moving cars), the authors note that explicitly handling dynamic components is a key direction for future work.
- Occlusion Resolution: The conservative disocclusion strategy can sometimes lead to "false negatives," where points are hidden that should be visible, suggesting room for a more sophisticated warping process.
Summary updated