Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey

Yang Tang, Chaoqiang Zhao, Jianrui Wang, Chongzhen Zhang, Qiyu Sun, Wei Xing Zheng, Wenli Du, Feng Qian, Jürgen Kurths

Abstract

Autonomous systems possess the features of inferring their own state, understanding their surroundings, and performing autonomous navigation. With the applications of learning systems, like deep learning and reinforcement learning, the visual-based self-state estimation, environment perception, and navigation capabilities of autonomous systems have been efficiently addressed, and many new learning-based algorithms have surfaced with respect to autonomous visual perception and navigation. In this review, we focus on the applications of learning-based monocular approaches in ego-motion perception, environment perception, and navigation in autonomous systems, which is different from previous reviews that discussed traditional methods. First, we delineate the shortcomings of existing classical visual simultaneous localization and mapping (vSLAM) solutions, which demonstrate the necessity to integrate deep learning techniques. Second, we review the visual-based environmental perception and understanding methods based on deep learning, including deep learning-based monocular depth estimation, monocular ego-motion prediction, image enhancement, object detection, semantic segmentation, and their combinations with traditional vSLAM frameworks. Then, we focus on the visual navigation based on learning systems, mainly including reinforcement learning and deep reinforcement learning. Finally, we examine several challenges and promising directions discussed and concluded in related research of learning systems in the era of computer science and robotics.

Introduction: The Transition to the Era of Learning

Autonomous systems are undergoing a fundamental shift from classical, geometry-heavy algorithms to learning-based paradigms. This survey provides a comprehensive roadmap of this transition, specifically examining how deep learning (DL) and reinforcement learning (RL) are solving the long-standing "monocular vision problem." By leveraging these technologies, robots can now perceive depth, estimate their own motion, and navigate complex environments using only a single camera.

The general architecture of an autonomous system, highlighting the interplay between perception, planning, and control.

Addressing the Limitations of Classical vSLAM

For decades, Visual Simultaneous Localization and Mapping (vSLAM) was the gold standard for robot autonomy. However, classical vSLAM has significant "Achilles' heels": it often fails in low-texture areas (such as plain white walls), struggles with dynamic lighting conditions (night, rain, or glare), and generally assumes a static world. Furthermore, geometric SLAM lacks semantic understanding—it perceives a cloud of points but cannot distinguish between a "car" that might move and a "road" that is drivable. Learning-based methods address these gaps by incorporating environmental priors and enabling end-to-end navigation that does not always require a pre-built map.

Pillars of Learning-Based Perception

The "learning revolution" in autonomous systems is built upon several key pillars:

  1. Monocular Depth Estimation: There is a significant move from supervised learning (requiring expensive LiDAR ground truth) to unsupervised learning. These models utilize "view reconstruction loss," learning that correct depth and motion predictions allow for the mathematical warping of one video frame into the next.
  2. Ego-motion Prediction: Using CNNs (like PoseNet) or RNNs, systems can directly regress the 6-DOF (degree of freedom) pose of a camera from image sequences, bypassing the need for manual feature matching.
  3. Hybrid Systems: Modern frameworks often integrate DL into traditional pipelines. For instance, a neural network might provide an initial "guess" for depth or pose, which is then refined by a classical geometric optimizer (e.g., CNN-SVO or DeepFactors).
Comparison of mapping outputs: (a) Sparse map, (b) Semi-dense map, (c) Dense map, and (d) Semantic map.

Visual Navigation via Deep Reinforcement Learning

Visual navigation is increasingly being driven by Deep Reinforcement Learning (DRL) techniques such as DQN, DDPG, A3C, and SAC. These methods map raw pixels directly to control signals, enabling "mapless" navigation. A notable subfield is Visual Language Navigation (VLN), where agents learn to follow natural language instructions, such as "go to the kitchen and find the apple," bridging the gap between high-level human intent and low-level robotic control.

Key Insights and Benchmarks

The field relies on several foundational benchmarks, including KITTI for outdoor driving, NYU Depth V2 for indoor depth estimation, and AI2-THOR for high-fidelity simulation.

A core insight is that estimating 3D structure from a single 2D image is an "ill-posed" problem without "priors." Deep learning provides these priors by learning what the world typically looks like. Furthermore, semantic segmentation acts as a filter; by identifying dynamic objects like buses or pedestrians, the system can ignore those pixels when calculating its own motion, greatly increasing robustness in traffic.

Challenges and Future Directions

Despite significant progress, several challenges remain:

  • Sim-to-Real Gap: DRL agents often perform perfectly in simulation but fail in the real world due to subtle physical and visual discrepancies.
  • Computational Efficiency: High-accuracy networks often require substantial GPU power, making real-time deployment on small drones or low-power mobile robots difficult.
  • Lifelong Learning: Most current systems are static after training. Future research must focus on systems that adapt and learn continuously as they encounter new environments.

Summary updated

Abstract

Autonomous systems possess the features of inferring their own state, understanding their surroundings, and performing autonomous navigation. With the applications of learning systems, like deep learning and reinforcement learning, the visual-based self-state estimation, environment perception, and navigation capabilities of autonomous systems have been efficiently addressed, and many new learning-based algorithms have surfaced with respect to autonomous visual perception and navigation. In this review, we focus on the applications of learning-based monocular approaches in ego-motion perception, environment perception, and navigation in autonomous systems, which is different from previous reviews that discussed traditional methods. First, we delineate the shortcomings of existing classical visual simultaneous localization and mapping (vSLAM) solutions, which demonstrate the necessity to integrate deep learning techniques. Second, we review the visual-based environmental perception and understanding methods based on deep learning, including deep learning-based monocular depth estimation, monocular ego-motion prediction, image enhancement, object detection, semantic segmentation, and their combinations with traditional vSLAM frameworks. Then, we focus on the visual navigation based on learning systems, mainly including reinforcement learning and deep reinforcement learning. Finally, we examine several challenges and promising directions discussed and concluded in related research of learning systems in the era of computer science and robotics.
BibTeX
@article{tang2022perception,
  author = {Yang Tang and Chaoqiang Zhao and Jianrui Wang and Chongzhen Zhang and Qiyu Sun and Wei Xing Zheng and Wenli Du and Feng Qian and Jürgen Kurths},
  title = {{Perception and Navigation in Autonomous Systems in the Era of Learning: A Survey}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2022},
  doi = {10.1109/tnnls.2022.3167688},
}
Figure