DIRT: Depth-completed Image-based Rendering using Two-band Blending for Live Novel-view Synthesis

Mathias Harrer, Laura Fink, Noah Lewis, Paul Himmler, Tim WeyricharXiv 2026 · arXiv

TL;DR

DIRT (Depth-completed Image-based Rendering with Two-band blending) is a non-neural, optimization-free pipeline that enables high-quality novel-view synthesis in milliseconds. By combining hierarchical sparse depth completion with frequency-aware blending, it allows for real-time scene exploration during capture across diverse modalities, including LiDAR, thermal, and handheld RGB-D sensors.

What Problem Does This Paper Address?

State-of-the-art novel-view synthesis (NVS) methods, such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS), typically require lengthy per-scene optimization—often taking minutes or hours—before they can render a scene from a new perspective. This makes them unsuitable for "live" applications like autonomous driving previews or interactive handheld scanning, where the delay between capture and rendering must be minimal.

While some "generalizable" neural networks attempt to skip optimization, they often struggle to generalize to new environments (e.g., switching from daylight to night-time or from RGB to thermal cameras). The authors address the need for a low-latency, sensor-agnostic NVS approach that doesn't rely on training data or scene-specific "baking."

Methods

DIRT returns to the roots of Image-Based Rendering (IBR) but modernizes it with a robust, non-neural pipeline. The process consists of three main stages:

  1. Geometric Proxy Construction: The system takes sparse point clouds (from LiDAR or RGB-D sensors) and projects them into the target novel view. To fill the resulting "holes," the authors use a hierarchical filling strategy. This recursive, image-space algorithm densifies the depth map while remaining anchored to actual measurements rather than "hallucinating" geometry like a neural network might.
  2. Warping and Disocclusion: Posed auxiliary images are warped onto this dense geometric proxy. To handle areas that should be occluded, the system renders the proxy as a triangle mesh from the perspective of the auxiliary cameras, performing a depth test to identify and mask out hidden regions.
  3. Two-Band Blending: To avoid the "ghosting" artifacts common when blending multiple images, DIRT employs a frequency-aware strategy. It decomposes images into high and low-frequency bands using a Laplacian Pyramid. Low frequencies (colors/lighting) are blended smoothly over a wide area, while high frequencies (sharp details) are selected from the most reliable source view. Blending weights are determined by a softEDT (approximate distance transform) to minimize seams near occlusion edges.
Figure 1: Pipeline architecture. Given posed images and depth observations, our method produces novel views in milliseconds. First, points are rendered to the target view’s screen space to construct a dense depth map through hierarchical depth completion. Auxiliary images are then warped onto the screen-space geometry, while resolving disocclusions. Finally, the images are blended using a two-band approach, weighted by an approximate distance transform as blending weight.

Datasets

The authors evaluated DIRT across a wide variety of challenging scenarios:

  • Automotive: nuScenes, Waymo, and Argoverse 2 (LiDAR + multi-camera setups).
  • Handheld RGB-D: The LiveNVS dataset.
  • Conventional NVS Benchmarks: MipNeRF 360 and Tanks and Temples (for offline comparison).
  • Cross-Modal: RGBT-Scenes (Thermal imagery) and night-time driving sequences.

Key Insights and Ideas

  • Non-Neural is a Feature, Not a Bug: By avoiding neural components, DIRT is inherently robust to "domain shift." It works on thermal data or low-light night scenes without needing to be retrained, whereas neural baselines often fail to converge or produce blurry results in these settings.
  • Measurement-Anchored Geometry: Unlike learned depth completion that might guess geometry, DIRT’s hierarchical filling stays faithful to the sensor data. This is critical for "capture guidance," where a user needs to see exactly what has (and hasn't) been recorded.
  • Frequency-Aware Blending: The "aha moment" for visual quality is the two-band blending. It allows the system to hide small misalignments in the geometry by smoothly transitioning the "glow" of the scene while keeping the high-frequency textures sharp.
Comparison of naive blending vs. two-band blending, illustrating the reduction in ghosting.

Key Results

  • Speed: DIRT achieves "capture-to-image" latencies of 69ms to 135ms, significantly faster than optimization-based methods that take seconds or minutes.
  • Perceptual Quality: In live reconstruction scenarios, DIRT consistently outperformed prior methods (like On-the-fly NVS and Long-LRM) in LPIPS, a metric that tracks human-perceived image quality.
  • Robustness: In night-time and thermal tests, DIRT maintained high fidelity while competing neural methods often failed to produce a consistent scene representation. For handheld RGB-D captures, DIRT improved PSNR by up to 11% over the next best state-of-the-art live method.

Limitations and Future Work

  • Edge Artifacts: While two-band blending helps, DIRT can still exhibit seam-like artifacts at depth discontinuities if the reconstructed geometry is slightly misaligned with the images.
  • Static Scene Assumption: The current pipeline assumes the world is static. While the point accumulation strategy filters out some transient objects (like moving cars), the authors note that explicitly handling dynamic components is a key direction for future work.
  • Occlusion Resolution: The conservative disocclusion strategy can sometimes lead to "false negatives," where points are hidden that should be visible, suggesting room for a more sophisticated warping process.

Summary updated

Abstract

State-of-the-art novel-view synthesis (NVS) methods commonly require per-scene optimization before rendering, limiting their applicability in live settings. Live or ad-hoc NVS, however, demands low-latency rendering during capture and strong generalization to new scenes. Some previous work tried to address this task by adapting training- and optimization-based methods, trading quality for low latency. We introduce a live NVS approach following traditional image-based rendering that operates on a posed set of depth sensor observations and images. Our method leverages explicit, sparse point-based geometry obtained from active sensors, resulting in a sparsely filled depth buffer that is completed in the (novel) target view's screen-space. Using this proxy, a posed set of images is projected into the target frame before fusing the resulting observations through a frequency-aware two-band blending strategy. The pipeline is entirely non-neural and optimization-free, requiring no training data or scene-specific optimization. As a result, our approach reduces the delay between capture and novel-view rendering and does not rely on favorable network generalization. We validate the method's applicability to heterogeneous sensor setups - including handheld RGB-D cameras, automotive LiDAR, and thermal cameras - as well as challenging scenarios, such as low light captures. Our experiments demonstrate that the approach outperforms prior methods in live reconstruction scenarios, while maintaining competitive performance in conventional NVS benchmarks.
BibTeX
@misc{harrer2026dirt,
  author = {Mathias Harrer and Laura Fink and Noah Lewis and Paul Himmler and Tim Weyrich},
  title = {{DIRT: Depth-completed Image-based Rendering using Two-band Blending for Live Novel-view Synthesis}},
  year = {2026},
  eprint = {2611.18340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18340},
}
Figure