ActNeRF: Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions

Saptarshi Dasgupta, Akshat Gupta, Shreshth Tuli, Rohan Paul

Abstract

Building complete 3D models of objects is a fundamental challenge for robotic manipulation, as the "bottom" of an object is typically occluded by the surface it rests upon. This paper proposes an active learning framework that utilizes an ensemble of Neural Radiance Fields (NeRFs) to quantify model uncertainty. This uncertainty guides a robot to perform both visual actions (moving the camera) and physical actions (flipping the object) to reveal and learn occluded surfaces. By integrating object segmentation and robust pose re-acquisition, the system achieves significant improvements in reconstruction accuracy and downstream manipulation success compared to traditional camera-only methods.

Introduction

To manipulate objects effectively in unstructured environments, robots require high-fidelity 3D models. While Neural Radiance Fields (NeRFs) have emerged as a powerful representation for 3D geometry and appearance, they are often limited by the viewpoints available to a static or mobile camera. Specifically, the base of an object remains hidden from line-of-sight observation. Existing "Next-Best-View" (NBV) algorithms focus on camera trajectory optimization but fail to address these physical occlusions. This work introduces a framework where the robot actively interacts with the object to "see" the unseen, bridging the gap between visual perception and physical manipulation.

The Uncertainty-Aware Pipeline

The proposed system follows a cyclic pipeline designed to minimize model uncertainty through strategic actions.

The proposed active learning pipeline, integrating NeRF training, uncertainty-driven action selection, and physical object re-orientation.

Key components of this pipeline include:

  • Ensemble-based Uncertainty: The system trains an ensemble of $M$ NeRF models with randomized initial weights. High variance in RGB rendering across the ensemble indicates regions of high uncertainty, signaling a need for more data.
  • Object-Centric Focus: To ensure the robot focuses on the object rather than the background, Grounded-SAM is used to segment and mask the object of interest. This prevents the active learning search from being distracted by background textures.
  • Bi-level Optimization for Action Selection: The robot evaluates potential actions (camera moves or object flips) by balancing the expected reduction in uncertainty against the operational cost (time and distance).
  • Uncertainty-Aware Grasping: To perform a flip, the system uses AnyGrasp to find candidates, which are then pruned based on the NeRF’s predicted material density and depth uncertainty to ensure a stable interaction.
  • Pose Re-acquisition: After a physical flip, the object's pose changes. The system employs a multi-strategy optimization approach to accurately re-estimate the new pose and integrate new observations into the existing model.

Key Insights and Results

The core innovation of this work is the demonstration that physical interaction is mandatory for model completeness. Quantitative results show that camera movement alone eventually plateaus in quality because it cannot resolve occlusions at the object's base.

Evolution of the NeRF model and uncertainty map as the robot captures new views and eventually flips the object to learn the previously hidden bottom surface.

Experimental evaluation in NVIDIA Isaac Sim using YCB dataset objects revealed:

  • Reconstruction Quality: A 14% improvement in PSNR and a 20% improvement in F-score for surface reconstruction.
  • Manipulation Success: A 71% increase in the success rate of downstream tasks where the robot had to manipulate objects in novel, previously unobserved orientations.
  • Efficiency: The uncertainty-aware strategy reached higher model fidelity with a lower total action cost compared to random or furthest-point sampling baselines.

Conclusion

This research highlights the necessity of physical interaction in the active learning of 3D models. By treating "flipping" as a valid exploratory action guided by NeRF ensemble uncertainty, robots can build truly complete object representations. Future work will focus on reducing the computational overhead of ensemble training and extending the framework to handle non-rigid and articulated objects in cluttered real-world scenes.

Summary updated

BibTeX
@inproceedings{dasgupta2024actnerf,
  author = {Saptarshi Dasgupta and Akshat Gupta and Shreshth Tuli and Rohan Paul},
  title = {{ActNeRF: Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801767},
}
Figure