NARUTO: Neural Active Reconstruction from Uncertain Target Observations

Ziyue Feng, Huangying Zhan, Zheng Chen, Qingan Yan, Xiangyu Xu, Changjiang Cai, Bing Li, Qilun Zhu, Yi Xu

TL;DR

NARUTO is the first neural active reconstruction system capable of operating with unrestricted 6-Degree-of-Freedom (6DoF) movement in large-scale indoor environments. By combining a fast hybrid neural representation with real-time, explicit uncertainty learning, the system autonomously plans paths toward poorly reconstructed areas, boosting environment reconstruction completeness from 73% to over 90%.

What Problem Does This Paper Address?

Active reconstruction is the challenge of commanding a robotic agent or camera to autonomously explore an unknown environment to build a high-fidelity 3D model. While implicit neural representations like Neural Radiance Fields (NeRFs) have revolutionized 3D reconstruction, their heavy computational demands have severely restricted their use in active vision.

Prior neural active mapping systems were forced to limit the agent's movement—constraining them to discrete "teleportation" locations, simple 2D planes, or small object-centric hemispheres. NARUTO breaks through these barriers, addressing the need for a real-time system that can handle continuous 6DoF exploration in large, complex, multi-room indoor spaces.

Methods

NARUTO integrates mapping, uncertainty quantification, and path planning into a unified, real-time loop:

The NARUTO framework overview, showing the pipeline from data generation to uncertainty-aware bundle adjustment and planning.
  1. Neural 3D Mapping: The system uses Co-SLAM as its backbone, which utilizes a multi-resolution hash-grid combined with coordinate encoding. This hybrid approach offers rapid convergence and excellent hole-filling capabilities.
  2. Explicit Uncertainty Learning: Instead of using an implicit Multi-Layer Perceptron (MLP) to guess uncertainty, NARUTO introduces a learnable Uncertainty Grid ($V_\sigma$). By tying uncertainty directly to depth rendering losses during bundle adjustment, the system dynamically maps out which parts of the 3D space are highly uncertain.
  3. Uncertainty-Aware Planning: The system aggregates the top-$k$ most uncertain points within the agent's sensing range to identify a target goal. It then utilizes an optimized path planner called Efficient RRT (E-RRT), which speeds up traditional 3D Rapidly-exploring Random Trees by actively attempting direct line-of-sight connections to the goal.
  4. Active Ray Sampling: During mapping optimization, rather than sampling pixels entirely at random, NARUTO uses its learned uncertainty map to actively sample rays from regions with poor reconstruction quality, acting as a plug-and-play booster for map stability.

Datasets

The authors evaluated NARUTO inside the Habitat Simulator using two photorealistic indoor 3D datasets:

  • Replica: A dataset of high-quality, dense digital replicas of indoor spaces (used primarily for ablation studies).
  • Matterport3D (MP3D): A large-scale dataset featuring complex, multi-room real-world indoor environments, presenting significant exploration challenges.

Key Insights and Ideas

  • Explicit Grids Beat Implicit Networks for Uncertainty: The authors discovered that using an MLP network to predict uncertainty introduces instability. Because an MLP's weights change globally during optimization, updates in one room can unpredictably alter the uncertainty values of unobserved rooms. An explicit 3D uncertainty volume ensures that updates remain strictly local.
  • Reachability Filtering is Essential: When aggregating uncertainty to find the next best view, the system uses its E-RRT planner to check if a high-uncertainty zone is actually reachable. If a zone is trapped behind a wall or outside the navigable boundaries, it is filtered out, preventing the agent from getting stuck.
Comparison of uncertainty decay and completion ratio between the explicit grid approach and an implicit network.

Key Results

  • State-of-the-Art Completeness: On the challenging MP3D dataset, NARUTO achieved a 90.18% completion ratio, drastically outperforming the previous state-of-the-art active mapping method (ANM), which sat at 73.15%.
  • High-Fidelity Geometry: NARUTO reduced the Mean Absolute Distance (MAD) error to just 1.44 cm compared to ANM's 4.29 cm, demonstrating much tighter alignment with ground-truth surfaces.
  • Real-Time Planning Speeds: Thanks to parallel GPU computing and direct-line shortcuts, the E-RRT path planner computes trajectories in just 5.77 milliseconds, compared to the agonizing 19 seconds required by traditional 3D RRT implementations.
Qualitative 3D reconstruction results on the Matterport3D dataset comparing ground truth, the baseline, and NARUTO.

Limitations and Future Work

While NARUTO marks a major step forward, a few limitations remain open for future research:

  • Assumption of Perfect Localization: The system currently assumes localization (tracking) is perfectly known and focuses entirely on the reconstruction aspect. In real-world robotics, noisy odometry must be handled jointly (Active SLAM).
  • Simplified Agent Physics: The agent is modeled as a simple, free-flying sphere. Real robots have non-holonomic motion constraints (e.g., wheeled robots can't instantly move sideways or fly).
  • Single-Resolution Uncertainty: The uncertainty grid uses a fixed voxel size (10cm). Transitioning to a multi-resolution uncertainty representation could help capture finer geometric details more efficiently.

Summary updated

Abstract

We present NARUTO, a neural active reconstruction system that combines a hybrid neural representation with uncertainty learning, enabling high-fidelity surface reconstruction. Our approach leverages a multi-resolution hash-grid as the mapping backbone, chosen for its exceptional convergence speed and capacity to capture high-frequency local features.The centerpiece of our work is the incorporation of an uncertainty learning module that dynamically quantifies reconstruction uncertainty while actively reconstructing the environment. By harnessing learned uncertainty, we propose a novel uncertainty aggregation strategy for goal searching and efficient path planning. Our system autonomously explores by targeting uncertain observations and reconstructs environments with remarkable completeness and fidelity. We also demonstrate the utility of this uncertainty-aware approach by enhancing SOTA neural SLAM systems through an active ray sampling strategy. Extensive evaluations of NARUTO in various environments, using an indoor scene simulator, confirm its superior performance and state-of-the-art status in active reconstruction, as evidenced by its impressive results on benchmark datasets like Replica and MP3D.
BibTeX
@misc{feng2024naruto,
  author = {Ziyue Feng and Huangying Zhan and Zheng Chen and Qingan Yan and Xiangyu Xu and Changjiang Cai and Bing Li and Qilun Zhu and Yi Xu},
  title = {{NARUTO: Neural Active Reconstruction from Uncertain Target Observations}},
  year = {2024},
  eprint = {2402.18771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.18771},
}
Figure