GoToNet: Fast Monocular Scene Exposure and Exploration

Tom Avrech, Evgenii Zheltonozhskii, Chaim Baskin, Ehud Rivlin

TL;DR

GoToNet is a lightweight, monocular navigation framework that enables drones to explore unknown environments in real-time using only a single forward-looking RGB camera. By predicting "Goto" and "Lookat" pixels, the system optimizes flight paths to maximize scene exposure, significantly outperforming existing methods in both exploration efficiency and computational speed.

What Problem Does This Paper Address?

Autonomous exploration in unknown, GPS-denied, or communication-denied environments (like indoor disaster zones or underground mines) remains a massive hurdle for robotics. Traditional Simultaneous Localization and Mapping (SLAM) methods are often too computationally heavy for Micro Aerial Vehicles (MAVs) and typically require expensive, heavy depth sensors like LiDAR.

Existing "Next-Best-View" (NBV) algorithms, which decide where a robot should look next to gather the most information, often rely on discrete action spaces (e.g., "turn 25 degrees left") and voxel-based maps that are slow to process. The authors of GoToNet sought a "one-look" approach: a system that can take a single RGB image and instantly decide the best tactical move to uncover the most "unseen" territory.

Methods

The GoToNet framework operates in a three-step deep learning pipeline:

  1. Depth Prediction: An RGB image is fed into a DenseDepth network (a UNet-like model with a DenseNet-161 encoder) to generate a depth map.
  2. Gradient Extraction: The system applies a Sobel-Feldman operator to the depth map. This highlights sharp changes in depth, which are strong indicators of open spaces or hidden areas behind objects.
  3. Navigation Instruction: A modified VGG16 network processes the gradient map to predict two specific sets of coordinates:
  4. Goto Pixel: Defines the 3D direction the agent should move.
  5. Lookat Pixel: Defines where the camera should point in the next step to maximize exposure.

The "secret sauce" lies in how they generated the training labels. In a simulator, they "painted" seen areas of a 3D scene. They then simulated a 3x3 grid of possible next poses, counted how many "unseen" pixels each would reveal, and used a weighted average of these results to determine the optimal Goto and Lookat coordinates.

Datasets

The researchers developed a novel dataset generator using the Microsoft AirSim simulator (built on Unreal Engine).

  • 3D Exploration Dataset: 48,400 data samples consisting of RGB images, corresponding depth maps, and optimized Goto/Lookat pixel coordinates.
  • 2D Exploration Dataset: 37,172 samples for a simplified horizontal-plane movement scenario.
  • Environment: A custom-made indoor scene featuring multiple rooms, diverse objects, and complex pathways to test navigation logic.

Key Insights and Ideas

  • Gradients over Raw Depth: The authors found that training on gradient maps is more effective than raw depth. Gradients explicitly signal "edges" and potential openings, preventing the model from overfitting on specific depth values and helping it identify "holes" in the scene that need exploring.
  • Continuous Action Space: Unlike many models that choose from a few fixed directions, GoToNet predicts pixel coordinates. This allows for much finer, more accurate movement and rotation adjustments.
  • The "Non-Exposing" Fail-Safe: If the drone faces a wall and no direction reveals new voxels (a "non-exposing case"), the model is trained to head toward the deepest point in the current depth map to find an exit.
  • Constant-Time Decision Making: Because the model only requires a single look to make a decision, its compute time doesn't grow as the map gets larger, making it ideal for long-duration missions.

Key Results

GoToNet was compared against "Learn-to-Score," a state-of-the-art voxel-based NBV method. The results were striking:

  • Exploration Efficiency: GoToNet exposed 850% more new voxels per camera pose than the baseline.
  • Target Acquisition: In tests aimed at finding a random target, GoToNet brought the drone 10.5% closer to the goal.
  • Speed: The mean step time on an RTX 2080 Max-Q was just 5.67 seconds, compared to 31.58 seconds for the baseline—a nearly 6x speedup.
  • Surface Coverage: GoToNet saw 8.89% of the total surface voxels in a trial, while the baseline only managed 0.89%.

Limitations and Future Work

The primary limitation acknowledged is the Sim-to-Real gap. While the AirSim environment is high-fidelity, transferring these models to physical drones involves dealing with real-world lighting, motion blur, and hardware constraints.

Future directions suggested by the authors include:

  • Cooperative Swarms: Developing multi-agent capabilities where drones share "seen" data to divide and conquer a scene.
  • End-to-End Optimization: Further shrinking the model size to reduce runtime even more.
  • Dense Sampling: Increasing the 3x3 grid used during training data generation to 9x9 or higher to potentially improve the "optimality" of the predicted paths.

Summary updated

Abstract

Autonomous scene exposure and exploration, especially in localization or communication-denied areas, useful for finding targets in unknown scenes, remains a challenging problem in computer navigation. In this work, we present a novel method for real-time environment exploration, whose only requirements are a visually similar dataset for pre-training, enough lighting in the scene, and an on-board forward-looking RGB camera for environmental sensing. As opposed to existing methods, our method requires only one look (image) to make a good tactical decision, and therefore works at a non-growing, constant time. Two direction predictions, characterized by pixels dubbed the Goto and Lookat pixels, comprise the core of our method. These pixels encode the recommended flight instructions in the following way: the Goto pixel defines the direction in which the agent should move by one distance unit, and the Lookat pixel defines the direction in which the camera should be pointing at in the next step. These flying-instruction pixels are optimized to expose the largest amount of currently unexplored areas. Our method presents a novel deep learning-based navigation approach that is able to solve this problem and demonstrate its ability in an even more complicated setup, i.e., when computational power is limited. In addition, we propose a way to generate a navigation-oriented dataset, enabling efficient training of our method using RGB and depth images. Tests conducted in a simulator evaluating both the sparse pixels’ coordinations inferring process, and 2D and 3D test flights aimed to unveil areas and decrease distances to targets achieve promising results. Comparison against a state-of-the-art algorithm shows our method is able to overperform it, that while measuring the new voxels per camera pose, minimum distance to target, percentage of surface voxels seen, and compute time metrics.
BibTeX
@article{avrech2022gotonet,
  author = {Tom Avrech and Evgenii Zheltonozhskii and Chaim Baskin and Ehud Rivlin},
  title = {{GoToNet: Fast Monocular Scene Exposure and Exploration}},
  journal = {J. Intell. Robotic Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=e1aMo-Q_q_d},
}

Loading related content...

Figure