Abstract
While training an end-to-end navigation network in the real world is usually of high cost, simulation provides a safe and cheap environment in this training stage. However, training neural network models in simulation brings up the problem of how to effectively transfer the model from simulation to the real world (sim-to-real). In this work, we regard the environment representation as a crucial element in this transfer process and propose a visual information pyramid (VIP) model to systematically investigate a practical environment representation. A novel representation composed of spatial and semantic information synthesis is then established accordingly, where noise model embedding is particularly considered. To explore the effectiveness of this representation, we compared the performance with representations popularly used in the literature in both simulated and real-world scenarios. Results suggest that our environment representation stands out. Furthermore, an analysis on the feature map is implemented to investigate the effectiveness through inner reaction, which could be irradiative for future researches on end-to-end navigation.
Introduction
Training mobile robots to navigate using end-to-end deep learning—mapping sensor data directly to motor commands—is a data-hungry process. While simulation offers a cheap and safe alternative for data gathering, models often struggle when deployed in the real world, a challenge known as the sim-to-real gap.
This research identifies the environment representation as a primary factor in this gap. Standard inputs like RGB images often contain simulation-specific textures that do not generalize, while raw simulated depth images are often "too perfect," lacking the sensor noise characteristic of real-world hardware like the Kinect. This paper proposes a representation that is abstract enough to generalize but detailed enough to ensure safety.
The Visual Information Pyramid (VIP)
The authors introduce the Visual Information Pyramid (VIP) model to categorize the types of information a robot requires for navigation. The model suggests that for robust navigation, the robot primarily needs two high-level information types: spatial (where objects are) and semantic (what objects are).
The VIP model prioritizes semantic and spatial information over lower-level textural and intensity data. Based on this, the researchers developed a representation that synthesizes:
- Spatial Information with Noise Embedding: Depth images are injected with a mathematical noise model (simulating Kinect V2) to include Gaussian noise on edges and random masked pixels.
- Categorized Risk Maps (Semantic): Using YOLO V3, objects are detected and mapped to six "collision risk levels" (e.g., pedestrians are level 5, furniture is level 2). This creates a grayscale image where intensity represents the danger level.
System Architecture
The proposed navigation system utilizes a dual-encoder architecture to process these synthesized inputs.
As shown in the architecture, separate CNN encoders process the noisy depth and the semantic risk maps. These features are concatenated with a global direction command (e.g., "turn left") and fed into a neural network that outputs linear and angular velocities. This approach was compared against three common baselines: pure RGB, raw depth, and standard semantic segmentation.
Experimental Results
The models were trained using expert "First-Person View" (FPV) operations in a Gazebo simulation and evaluated in both complex simulated environments and a real-world 90-meter cluttered corridor.
The proposed model, $E_{Dep.Noi.Det.}$, significantly outperformed all baselines:
- Simulation: It required only 0.2 interventions per trial, compared to 2.5 for RGB and 5.6 for standard depth.
- Real-World: In the corridor test, it averaged 0.8 interventions per trial. In contrast, RGB and standard depth models were essentially non-functional, requiring constant human assistance (over 20 interventions).
- Reactive Behavior: The model demonstrated a clear "reaction distance," slowing down significantly when a pedestrian was within 2 meters—a behavior that was much less stable in the baseline models.
Key Insights
- Less is More: RGB images can be detrimental to sim-to-real transfer because the network overfits to simulation-specific textures.
- Robustness through Noise: Intentionally adding noise to simulated depth maps makes the network robust to real-world sensor artifacts.
- Risk-Based Semantics: Grading objects by risk allows the network to prioritize safety, such as giving a wider berth to pedestrians than to stationary furniture.
Conclusion
By focusing on a high-level representation of spatial and semantic information, the researchers successfully bridged the sim-to-real gap for indoor navigation. Future work may involve moving from imitation learning to Reinforcement Learning (RL) to further optimize recovery maneuvers and testing in more chaotic, unstructured environments.
Summary updated