Ternary-Type Opacity and Hybrid Odometry for RGB NeRF-SLAM

Junru Lin, Asen Nachkov, Songyou Peng, Luc Van Gool, Danda Pani Paudel

Abstract

Simultaneous Localization and Mapping (SLAM) using Neural Radiance Fields (NeRFs) has traditionally relied on depth-sensing hardware to achieve high-fidelity reconstructions. In the absence of depth sensors, RGB-only NeRF-SLAM faces significant challenges, primarily "foggy" geometric reconstructions and high computational latency. This paper presents a novel framework that addresses these issues through two key innovations: a Ternary-Type Opacity (TT) model and a Hybrid Odometry (HO) scheme. Our approach achieves state-of-the-art accuracy on standard benchmarks and a 6x speedup over existing RGB-only baselines.

1. Introduction

NeRF-based SLAM has emerged as a powerful paradigm for dense 3D reconstruction. However, the reliance on RGB-D sensors limits its applicability in outdoor environments or on hardware where active depth sensing is unavailable. Transitioning to RGB-only inputs introduces two major hurdles. First, without explicit depth supervision, the volumetric density optimization often results in "cloudy" or "foggy" geometry, where opacity is distributed along a ray rather than concentrated on a physical surface. Second, the computational cost of tracking camera poses via full volumetric rendering is prohibitive for real-time applications.

This work proposes a solution that enforces physical priors about surface opacity and optimizes the tracking pipeline to bridge the gap between RGB-only and RGB-D performance.

2. Methodology

2.1 Ternary-Type Opacity (TT)

To resolve the ambiguity of depth in RGB-only settings, we introduce a Ternary-Type Opacity model. Standard NeRFs often struggle to distinguish between empty space and occluded regions, leading to blurred surfaces. Our model categorizes points along a camera ray into three distinct regions:

  1. Pre-surface: Points before the object surface, which are constrained to zero opacity.
  2. Surface: Points on the actual surface, which are optimized for high opacity.
  3. Post-surface: Points behind the surface that are occluded from the current viewpoint.

By utilizing a "softly-binarized" decoder, the system is forced to converge toward sharp, well-defined surfaces. This ternary approach acknowledges that points behind a surface are unobserved and should not negatively influence the optimization of the visible geometry, effectively eliminating the "foggy" artifacts common in previous methods.

2.2 Hybrid Odometry (HO)

To address the latency bottleneck, we move away from the traditional approach of rendering full volumetric images for every tracking frame. Instead, we implement a two-stage Hybrid Odometry scheme:

  • Coarse Tracking (Gradient-based Warping): We utilize a fast image-warping technique that projects point clouds from previous frames onto the current frame to minimize photometric reprojection error. This step provides a rapid initial pose estimate without the need for expensive NeRF rendering.
  • Fine Adjustment (Bundle Adjustment): Periodically, the system performs joint Bundle Adjustment (BA) on selected keyframes. This process refines both the camera poses and the underlying feature grids (the 3D scene representation) to ensure global consistency.

3. Experimental Results

3.1 Datasets and Performance

We evaluated our framework on the Replica synthetic dataset and the 7-Scenes real-world dataset. The latter is particularly challenging due to motion blur and textureless surfaces typical of handheld camera captures.

Our method demonstrated a significant improvement in both speed and precision:

  • Efficiency: The system achieved a 6x speedup compared to DIM-SLAM, a leading RGB-only competitor.
  • Accuracy: On the Replica dataset, our method achieved an Absolute Trajectory Error (ATE RMSE) as low as 0.59cm in certain scenes, outperforming DIM-SLAM's 0.86cm.
  • Mapping Quality: The combination of the TT model and HO scheme resulted in significantly sharper depth maps and higher PSNR scores, indicating superior image reconstruction quality.

3.2 Synergy of Innovations

A critical finding of our research is the synergy between the mapping and tracking components. The improved geometry produced by the Ternary-Type Opacity model directly enhances the accuracy of the image warping in the Hybrid Odometry scheme. This creates a virtuous cycle where more accurate mapping facilitates faster and more reliable tracking.

4. Conclusion and Future Work

This paper demonstrates that high-quality, efficient SLAM is achievable using only RGB inputs by incorporating physical opacity priors and a hybrid tracking pipeline. While our system provides a 6x speedup, it is not yet fully real-time on consumer-grade hardware, currently requiring approximately 145ms for bundle adjustment on an NVIDIA A100 GPU.

Future research will focus on hardware-specific code optimizations to reach true real-time performance and extending the system's capabilities to handle large-scale outdoor environments and complex materials, such as transparent or reflective surfaces.

Summary updated

Abstract

In this work, we address the challenge of deploying Neural Radiance Field (NeRFs) in Simultaneous Localization and Mapping (SLAM) under the condition of lacking depth information, relying solely on RGB inputs. The key to unlocking the full potential of NeRF in such a challenging context lies in the integration of real-world priors. A crucial prior we explore is the binary opacity prior of 3D space with opaque objects. To effectively incorporate this prior into the NeRF framework, we introduce a ternary-type opacity (TT) model instead, which categorizes points on a ray intersecting a surface into three regions: before, on, and behind the surface. This enables a more accurate rendering of depth, subsequently improving the performance of image warping techniques. Therefore, we further propose a novel hybrid odometry (HO) scheme that merges bundle adjustment and warping-based localization. Our integrated approach of TT and HO achieves state-of-the-art performance on synthetic and real-world datasets, in terms of both speed and accuracy. This breakthrough underscores the potential of NeRF-SLAM in navigating complex environments with high fidelity.
BibTeX
@inproceedings{lin2024ternary,
  author = {Junru Lin and Asen Nachkov and Songyou Peng and Luc Van Gool and Danda Pani Paudel},
  title = {{Ternary-Type Opacity and Hybrid Odometry for RGB NeRF-SLAM}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802493},
}
Figure