GoToNet: Fast Monocular Scene Exposure and Exploration

Tom Avrech, Evgenii Zheltonozhskii, Chaim Baskin, Ehud Rivlin

Abstract

Autonomous exploration in unknown environments remains a significant challenge for robotics, particularly in localization- or communication-denied areas. We present GoToNet, a novel deep learning-based navigation framework that enables real-time environment exploration using only a single forward-looking RGB camera. By predicting "Goto" and "Lookat" pixels, our method optimizes flight paths to maximize scene exposure in constant time. Experimental results in simulated environments demonstrate that GoToNet significantly outperforms state-of-the-art methods in exploration efficiency, target acquisition, and computational speed.

1. Introduction

Autonomous exploration in unknown, GPS-denied, or communication-denied environments—such as indoor disaster zones or underground mines—is a critical capability for modern robotics. However, traditional Simultaneous Localization and Mapping (SLAM) methods are often too computationally expensive for Micro Aerial Vehicles (MAVs) and typically require heavy, power-hungry depth sensors like LiDAR.

Existing "Next-Best-View" (NBV) algorithms attempt to solve this by deciding where a robot should look next to gather the most information. However, these often rely on discrete action spaces and voxel-based maps that scale poorly with environment size. GoToNet addresses these limitations by introducing a "one-look" approach: a system that uses a single RGB image to instantly determine the optimal tactical move to uncover the maximum amount of unseen territory.

2. The GoToNet Framework

The core of GoToNet is a three-step deep learning pipeline designed for speed and efficiency. Unlike methods that require complex 3D reconstructions before making a decision, GoToNet processes visual data directly into navigation commands.

2.1 Depth Prediction and Gradient Extraction

The process begins by feeding a single RGB image into a DenseDepth network. This model utilizes a UNet-like architecture with a DenseNet-161 encoder to generate a high-fidelity depth map. To make the model more robust to specific environment geometries, the system applies a Sobel-Feldman operator to the resulting depth map. This extracts depth gradients, highlighting sharp changes that indicate open spaces or hidden areas behind foreground objects. The authors found that training on these gradient maps prevents the model from overfitting on specific depth values and helps it identify "holes" in the scene that require exploration.

2.2 Navigation Instruction: Goto and Lookat Pixels

The final stage of the pipeline uses a modified VGG16 network to process the gradient map and predict two specific sets of coordinates:

  • Goto Pixel: Defines the 3D direction the agent should move by one distance unit.
  • Lookat Pixel: Defines the direction the camera should point in the next step to maximize future exposure.

By predicting pixel coordinates rather than choosing from a fixed set of directions (e.g., "left" or "right"), GoToNet operates in a continuous action space, allowing for much finer and more accurate movement adjustments.

3. Training and Data Generation

To train GoToNet, the researchers developed a custom dataset generator using the Microsoft AirSim simulator. They created a diverse indoor environment featuring complex pathways and various objects.

The training labels were generated using a "look-ahead" strategy. In the simulator, the system "painted" seen areas of a 3D scene. It then evaluated a 3x3 grid of possible next poses, counting how many previously unseen voxels each pose would reveal. A weighted average of these results determined the optimal Goto and Lookat coordinates for the training set. The final dataset included over 48,000 samples for 3D exploration and 37,000 samples for 2D horizontal-plane scenarios.

4. Experimental Results

GoToNet was evaluated against "Learn-to-Score," a state-of-the-art voxel-based NBV method. The performance metrics focused on exploration efficiency and computational overhead.

  • Exploration Efficiency: GoToNet exposed 850% more new voxels per camera pose than the baseline. It also achieved a surface coverage of 8.89%, compared to the baseline's 0.89%.
  • Target Acquisition: In missions aimed at finding a specific target, GoToNet brought the drone 10.5% closer to the goal than the competing method.
  • Computational Speed: On an RTX 2080 Max-Q, GoToNet achieved a mean step time of 5.67 seconds, nearly six times faster than the baseline's 31.58 seconds. Crucially, because the model only requires a single image to make a decision, its compute time remains constant regardless of the map's total size.

5. Conclusion

GoToNet demonstrates that high-efficiency autonomous exploration is possible using limited hardware and monocular vision. By shifting from complex map-based reasoning to a direct "one-look" pixel prediction model, the framework enables MAVs to navigate unknown environments with unprecedented speed. While future work is needed to bridge the sim-to-real gap and implement multi-agent cooperative swarms, GoToNet provides a robust foundation for the next generation of autonomous exploration robots.

Summary updated

Abstract

Autonomous scene exposure and exploration, especially in localization or communication-denied areas, useful for finding targets in unknown scenes, remains a challenging problem in computer navigation. In this work, we present a novel method for real-time environment exploration, whose only requirements are a visually similar dataset for pre-training, enough lighting in the scene, and an on-board forward-looking RGB camera for environmental sensing. As opposed to existing methods, our method requires only one look (image) to make a good tactical decision, and therefore works at a non-growing, constant time. Two direction predictions, characterized by pixels dubbed the Goto and Lookat pixels, comprise the core of our method. These pixels encode the recommended flight instructions in the following way: the Goto pixel defines the direction in which the agent should move by one distance unit, and the Lookat pixel defines the direction in which the camera should be pointing at in the next step. These flying-instruction pixels are optimized to expose the largest amount of currently unexplored areas. Our method presents a novel deep learning-based navigation approach that is able to solve this problem and demonstrate its ability in an even more complicated setup, i.e., when computational power is limited. In addition, we propose a way to generate a navigation-oriented dataset, enabling efficient training of our method using RGB and depth images. Tests conducted in a simulator evaluating both the sparse pixels' coordinations inferring process, and 2D and 3D test flights aimed to unveil areas and decrease distances to targets achieve promising results. Comparison against a state-of-the-art algorithm shows our method is able to overperform it, that while measuring the new voxels per camera pose, minimum distance to target, percentage of surface voxels seen, and compute time metrics.
BibTeX
@article{avrech2022gotonet,
  author = {Tom Avrech and Evgenii Zheltonozhskii and Chaim Baskin and Ehud Rivlin},
  title = {{GoToNet: Fast Monocular Scene Exposure and Exploration}},
  journal = {J. Intell. Robotic Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=e1aMo-Q_q_d},
}
Figure