Rendering the Directional TSDF for Tracking and Multi-Sensor Registration with Point-To-Plane Scale ICP

Malte Splietker, Sven Behnke

Abstract

This paper presents a novel framework for utilizing the Directional Truncated Signed Distance Function (DTSDF) in real-time 3D mapping and tracking. By storing surface information in six cardinal directions, the DTSDF effectively resolves geometry corruption in thin objects—a common failure mode for standard TSDFs. We introduce a rendering pipeline that selectively combines directional volumes for conflict-free view generation and a Sim(3) point-to-plane ICP algorithm to handle scale discrepancies in multi-sensor registration.

Introduction

Standard 3D reconstruction techniques, such as KinectFusion, rely on Truncated Signed Distance Functions (TSDFs) to represent surfaces. While effective for large structures, TSDFs struggle with thin objects where the front and back surfaces reside within the same voxel's truncation distance. This leads to "eroded" geometry and tracking drift.

Visualization of the DTSDF mapping approach, where different colors represent different surface normal directions stored within the voxel grid.

The Directional TSDF (DTSDF) was previously proposed to store these directional measurements separately. However, its adoption has been limited by the lack of efficient rendering methods for real-time tracking. This work bridges that gap, making DTSDF a viable "drop-in" replacement for standard mapping systems.

Methodology

The proposed system integrates three core technical advancements to enable robust, real-time robotic mapping:

1. DTSDF Rendering and Color Fusion

To enable tracking, the system must "render" a virtual view from the stored DTSDF. The authors developed a raycasting method that selectively combines the six directional volumes based on the camera's viewpoint. By ignoring directional data facing away from the camera (the "backside"), the system generates clean, conflict-free depth and color images.

The system pipeline: DTSDF volumes are combined based on pose to render depth and RGB maps, which are then used for ICP tracking.

2. Frame-to-Keyframe (f2kf) Tracking

To balance resolution and drift, the authors introduce frame-to-keyframe photometric tracking. Instead of tracking against the entire (potentially blurry) model or just the previous frame, the current RGB-D frame is compared to a high-quality keyframe, ensuring stable alignment and reducing the "blurring" effect common in frame-to-model tracking.

3. Sim(3) Point-to-Plane ICP

In multi-sensor environments, such as agricultural robots using both LiDAR and stereo cameras, calibration errors often manifest as scale differences. The authors extended standard SE(3) ICP to Sim(3), incorporating a scale parameter that allows the system to align point clouds from disparate sensors by adjusting their relative scale.

Experimental Results

The algorithm was validated on standard benchmarks (TUM RGB-D, ICL-NUIM) and a specialized agricultural dataset (PhenoRob) featuring sugar beet crops.

  • Tracking Performance: DTSDF significantly improved accuracy in scenes with thin structures. In the "Zhou office" sequence, the Relative Pose Error (RPE) was reduced by nearly 50% compared to standard TSDF methods at larger voxel sizes.
  • Memory Efficiency: Despite storing six directions, voxel hashing keeps memory usage within 1.5x to 2x of a standard TSDF, as most voxels only contain data from a few directions.
  • Multi-Sensor Registration: The Sim(3) optimization proved essential for high-precision mapping in agricultural settings, successfully resolving scale discrepancies between different depth-sensing technologies.

Conclusion

By providing the necessary rendering math and improved tracking techniques, this work transforms the DTSDF from a theoretical storage format into a practical tool for robust robotic mapping. While the memory overhead is manageable on modern GPUs, future work will focus on further optimization for low-power mobile platforms and extending the system to handle dynamic environments.

Summary updated

Abstract

Dense real-time tracking and mapping from RGB-D images is an important tool for many robotic applications, such as navigation and manipulation. The recently presented Directional Truncated Signed Distance Function (DTSDF) is an augmentation of the regular TSDF that shows potential for more coherent maps and improved tracking performance. In this work, we present methods for rendering depth- and color images from the DTSDF, making it a true drop-in replacement for the regular TSDF in established trackers. We evaluate the algorithm on well-established datasets and observe that our method improves tracking performance and increases re-usability of mapped scenes. Furthermore, we add color integration which notably improves color-correctness at adjacent surfaces. Our novel formulation of combined ICP with frame-to-keyframe photometric error minimization further improves tracking results. Lastly, we introduce Sim3 point-to-plane ICP for refining pose priors in a multi-sensor scenario with different scale factors.
BibTeX
@article{splietker2023rendering,
  author = {Malte Splietker and Sven Behnke},
  title = {{Rendering the Directional TSDF for Tracking and Multi-Sensor Registration with Point-To-Plane Scale ICP}},
  journal = {Robotics Auton. Syst.},
  year = {2023},
  doi = {10.1016/j.robot.2022.104337},
}
Figure