Real-time Neural Rendering of LiDAR Point Clouds

Joni Vanherck, Brent Zoomers, Tom Mertens, Lode Jorissen, Nick Michiels

TL;DR

This paper introduces a real-time neural rendering pipeline that transforms raw LiDAR point clouds into photorealistic images without requiring scene-specific training. By combining a novel depth-based heuristic filter with a U-Net reconstruction model, the authors achieve high-quality results that are over 2,000 times faster than previous state-of-the-art methods.

What Problem Does This Paper Address?

LiDAR scanners provide incredibly accurate 3D spatial data, but rendering that data for human viewing is surprisingly difficult. Raw point clouds are often "noisy" and incomplete. When you try to project points directly onto a 2D screen, you run into three major issues:

  1. Background Leakage: Because point clouds are sparse, background points often "peek through" gaps in foreground objects, creating a messy, unintelligible "screen door" effect.
  2. Incomplete Coverage: Scanners have blind spots and occlusions, leading to holes in the geometry.
  3. Artifacts: Merging scans from different positions often results in color inconsistencies and varying point densities.

Existing solutions usually require expensive preprocessing (like building meshes) or training a new neural network for every single scene, which is impractical for real-time applications or large-scale datasets.

Methods

The authors propose a four-stage pipeline designed for speed and output quality:

  1. Point Cloud Projection: Points are projected as simple 1x1 pixels. This is extremely fast and preserves maximum detail compared to using larger "splats," which tend to blur the image.
  2. Depth-Based Filtering: To solve the background leakage problem, the authors developed a heuristic based on hierarchical min-pooling. By comparing depth values across different resolutions, the algorithm identifies and discards points that are likely part of the background leaking through foreground gaps.
  3. U-Net Reconstruction: The filtered image (containing RGB, Depth, and an Alpha mask) is fed into a convolutional U-Net. This network handles the "heavy lifting": inpainting missing gaps, smoothing out color inconsistencies, and denoising the final output.
  4. Synthetic Training Data: To train the U-Net effectively, the authors created a synthetic dataset. They took high-quality photos and "damaged" them to look like raw LiDAR projections. This ensured the network had perfectly aligned ground-truth data to learn from, which is rarely available in real-world LiDAR/camera setups.
Figure 1: This figure provides an overview of the proposed method. A space-partitioned point cloud is projected onto the camera view using frustum culling, generating RGB, depth (D) and binary fill mask (A). The depth channel is used for depth filtering, as detailed in Fig. 2. A fully convolutional U-Net then processes the resulting depth-filtered RGBDA image to produce the final output.

Datasets

The researchers utilized the ScanNet++ dataset, which contains high-fidelity LiDAR scans and registered RGB images of indoor scenes.

  • ScanNet++ v1: Used for training the U-Net (approx. 19,000 image pairs).
  • ScanNet++ v2: Used for final evaluation on 55 unseen scenes to test the model's ability to generalize to new environments without retraining.

Key Insights and Ideas

  • Filtering is better than Splatting: Traditional point rendering uses large circular "splats" to fill gaps. The authors found that keeping points as 1x1 pixels and using a neural network to fill the gaps produces much sharper results.
  • Pre-filtering helps the AI: While a U-Net is powerful, it struggles if the input is cluttered with background noise. The depth-based heuristic "cleans" the input, allowing the U-Net to focus on reconstruction rather than sorting out depth layers.
  • Scene-Agnosticism: By training on a diverse synthetic dataset, the model learns general rules about how LiDAR data looks, meaning it can render a brand-new room it has never seen before in real-time.
The hierarchical structure of the depth-based filter, which uses min-pooling to distinguish foreground from background.

Key Results

The method was tested against Pointersect, a recent high-quality neural renderer.

  • Speed: The proposed method achieved 13.63 FPS at 1920x1440 and nearly 50 FPS at 960x720 on an NVIDIA RTX 4090. This is roughly 2,200x faster than Pointersect, which takes minutes to render a single frame.
  • Quality: Despite the massive speedup, the method outperformed the baseline in standard metrics like PSNR (16.10 vs 14.94) and SSIM (0.74 vs 0.60), producing images with fewer artifacts and better hole-filling.
Visual comparison showing how the method handles low-density (10%) and high-density (100%) point clouds compared to other techniques.

Limitations and Future Work

While the results are impressive, the authors note a few remaining challenges:

  • Complex Materials: The system still struggles with reflective or transparent surfaces (like mirrors or glass), which are notoriously difficult for LiDAR to capture accurately.
  • Dynamic Scenes: The current model is designed for static environments; moving objects would likely introduce significant artifacts.
  • Large-scale Defects: Very large gaps in the point cloud (beyond what a standard U-Net can "imagine") still result in visible blurring or incorrect inpainting.

Summary updated

Abstract

Static LiDAR scanners produce accurate, dense, colored point clouds, but often contain obtrusive artifacts which makes them ill-suited for direct display. We propose an efficient method to render photorealistic images of such scans without any expensive preprocessing or training of a scene-specific model. A naive projection of the point cloud to the output view using 1x1 pixels is fast and retains the available detail, but also results in unintelligible renderings as background points leak in between the foreground pixels. The key insight is that these projections can be transformed into a realistic result using a deep convolutional model in the form of a U-Net, and a depth-based heuristic that prefilters the data. The U-Net also handles LiDAR-specific problems such as missing parts due to occlusion, color inconsistencies and varying point densities. We also describe a method to generate synthetic training data to deal with imperfectly-aligned ground truth images. Our method achieves real-time rendering rates using an off-the-shelf GPU and outperforms the state-of-the-art in both speed and quality.
BibTeX
@inproceedings{vanherck2025real,
  author = {Joni Vanherck and Brent Zoomers and Tom Mertens and Lode Jorissen and Nick Michiels},
  title = {{Real-time Neural Rendering of LiDAR Point Clouds}},
  booktitle = {Eurographics (short Papers)},
  year = {2025},
}
Figure