Ternary-Type Opacity and Hybrid Odometry for RGB NeRF-SLAM

Junru Lin, Asen Nachkov, Songyou Peng, Luc Van Gool, Danda Pani Paudel

TL;DR

This paper introduces a novel RGB-only NeRF-SLAM framework that achieves state-of-the-art accuracy and a 6x speedup over existing methods. By incorporating a ternary-type opacity (TT) prior to better model opaque surfaces and a hybrid odometry (HO) scheme that avoids expensive volumetric rendering for every frame, the authors bridge the gap between high-fidelity neural scene representation and efficient real-time tracking.

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) is a cornerstone of robotics and AR/VR. While Neural Radiance Fields (NeRFs) have revolutionized 3D reconstruction, their integration into SLAM—especially using only RGB cameras—remains challenging.

Without explicit depth sensors (RGB-D), NeRF-SLAM systems often struggle with two main issues:

  1. Geometric Ambiguity: It is difficult to recover sharp, accurate surfaces from color alone. Standard NeRFs often produce "foggy" or semi-transparent geometry because there is no strong prior to force space to be either empty or solid.
  2. Computational Cost: Tracking a camera by performing full volumetric rendering (VR) for every frame is extremely slow, making real-time performance a distant goal for most monocular NeRF-SLAM methods.

Methods

The authors propose two primary innovations to tackle these hurdles:

1. Ternary-Type Opacity (TT)

In a real-world scene with opaque objects, a ray of light travels through empty space (zero opacity), hits a surface (high opacity), and then... we don't know. Standard "binary" priors try to force everything to be either 0 or 1. However, the authors argue that points behind the first opaque surface don't contribute to the image and shouldn't be forced to a specific value during optimization.

As shown in Figure 2, they use multi-resolution feature grids to infer opacity. Their TT model categorizes points along a ray into three regions: before (empty), on (surface), and behind (irrelevant). By fixing the "behind" points to a meaningful initial value ($o_{init}$) rather than letting them fluctuate, they achieve much sharper weight distributions and more accurate depth rendering.

Fig. 2: Inferring the Opacity of a 3D Point. We utilize a set of multi-resolution feature grids which we interpolate at the desired 3D point. The collected features are passed to a neural network to predict the color and the opacity. Only the opacity is shown for clarity.

2. Hybrid Odometry (HO)

To solve the speed problem, the authors move away from frame-wise volumetric rendering for tracking. Instead, they use a two-stage process illustrated in Figure 5:

  • Gradient-based Localization (GL): For coarse tracking, they project a point cloud from previous frames onto the current frame and minimize reprojection error. This "warping" approach is significantly faster than rendering a full NeRF image.
  • Bundle Adjustment (BA): Finer adjustments are made periodically by jointly optimizing camera poses and the neural scene representation using a subset of keyframes.
Fig. 5: Hybrid Odometry. For simplicity t refers to the index of the first frame from the current frame group Gm and T1 is the first frame from the tracking frames. In gradient-based localization by warping, we initialize poses by constant velocity, render the depth for some pixels on the last few frames T from the previous frame group to get point cloud X, and update the camera poses by minimizing the reprojection loss of X to the current frame Gm. In bundle adjustment we select several keyframes and together with Gm use VR to produce pixel values. The loss is backpropagated to the feature grids and the camera poses belonging to frames in Gm.

Datasets

The method was evaluated on two standard benchmarks:

  • Replica: A synthetic dataset of high-quality indoor room scans.
  • 7-Scenes: A challenging real-world dataset featuring handheld camera motion in indoor environments.

Key Insights and Ideas

  • The "Behind the Surface" Problem: The most significant "aha" moment is the realization that forcing a binary prior on unobserved regions (behind surfaces) hinders optimization. The ternary model (Fig. 3) allows the network to focus its "energy" on the actual surface, leading to a much higher peak in the weight distribution at the correct depth.
  • Decoupling Tracking from Rendering: By using image warping for tracking (GL) and reserving volumetric rendering for mapping (BA), the system achieves a massive speed boost without sacrificing the benefits of the neural representation.
  • Soft Binarization: The use of a softly-binarized decoder with a tunable temperature parameter allows the system to transition from a continuous representation to a sharp, opaque one during the initialization phase.
Fig. 3: Opacity and Weights along a Ray. With the ternary- type opacity (TT), the weights along a randomly sampled ray are more concentrated near the depth with a higher peak. The dots on the curves represent the sampled points on the ray. Data obtained from Replica [11] office-0 at the end of the training.

Key Results

The proposed method demonstrates a clear lead over the baseline DIM-SLAM across several metrics:

  • Speed: Achieves a 6x speedup compared to DIM-SLAM (see Fig. 1).
  • Tracking Accuracy: On the Replica dataset, it achieved an ATE RMSE as low as 0.59cm (office-0), significantly lower than the baseline's 0.86cm.
  • Mapping Quality: The ternary prior leads to much cleaner depth maps and higher PSNR (photometric quality). As shown in Figure 6, the method maintains high performance even with fewer Bundle Adjustment iterations, proving its robustness.
Fig. 1: Qualitative and Quantitative Results. On the left, we show the rendered RGB and depth from a random pose after training the whole sequence of Replica [11] room-1. On the right, we show the speed, tracking error, and mapping error on Replica office-0. DIM-SLAM∗refers to our re- implementation for DIM-SLAM [8].

Limitations and Future Work

Despite the impressive 6x speedup, the authors acknowledge that the system is not yet real-time on consumer-grade hardware. The current implementation relies on powerful GPUs (like the NVIDIA A100) to achieve its performance.

Future work will likely focus on:

  • Hardware-specific optimizations: Leveraging specialized kernels to further reduce the latency of the feature grid interpolations.
  • Large-scale environments: Testing how the ternary prior holds up in much larger, more complex outdoor scenes where the "opaque surface" assumption might be challenged by thin structures or transparency.

Summary updated

Abstract

In this work, we address the challenge of deploying Neural Radiance Field (NeRFs) in Simultaneous Localization and Mapping (SLAM) under the condition of lacking depth information, relying solely on RGB inputs. The key to unlocking the full potential of NeRF in such a challenging context lies in the integration of real-world priors. A crucial prior we explore is the binary opacity prior of 3D space with opaque objects. To effectively incorporate this prior into the NeRF framework, we introduce a ternary-type opacity (TT) model instead, which categorizes points on a ray intersecting a surface into three regions: before, on, and behind the surface. This enables a more accurate rendering of depth, subsequently improving the performance of image warping techniques. Therefore, we further propose a novel hybrid odometry (HO) scheme that merges bundle adjustment and warping-based localization. Our integrated approach of TT and HO achieves state-of-the-art performance on synthetic and real-world datasets, in terms of both speed and accuracy. This breakthrough underscores the potential of NeRF-SLAM in navigating complex environments with high fidelity.
BibTeX
@inproceedings{lin2024ternary,
  author = {Junru Lin and Asen Nachkov and Songyou Peng and Luc Van Gool and Danda Pani Paudel},
  title = {{Ternary-Type Opacity and Hybrid Odometry for RGB NeRF-SLAM}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802493},
}
Figure