Mapless Humanoid Navigation Using Learned Latent Dynamics

André Brandenburger, Diego Rodriguez, Sven Behnke

Researchers from the Autonomous Intelligent Systems Group at the University of Bonn have developed a robust approach to a classic robotics challenge: enabling a humanoid robot to navigate unknown environments without a pre-existing map. By combining "imagination-based" planning with a novel failure prediction mechanism, the team successfully transferred a controller trained entirely in simulation to a real NimbRo-OP2X humanoid robot.

Abstract

This work introduces a mapless navigation system for humanoid robots that utilizes a learned world model to plan paths by imagining future outcomes in a compressed latent space. By integrating visual data with internal sensor measurements and a novel "termination likelihood" predictor, the system achieves high sample efficiency and successfully transfers skills from simulation to real-world hardware. The approach demonstrates an 86% success rate in complex simulated environments and real-time navigation on physical hardware.

Introduction

Navigation for humanoid robots is notoriously difficult. Unlike wheeled platforms, humanoids experience significant camera "bobbing" due to their gait, possess a high center of mass that makes them prone to falling, and require complex, high-dimensional movement control. Traditional navigation relies on building a map via SLAM (Simultaneous Localization and Mapping), which is computationally expensive and often fails in dynamic environments.

Mapless navigation using Deep Reinforcement Learning (DRL) offers a promising alternative but typically requires massive amounts of data. Furthermore, while most existing mapless systems rely on depth sensors like LiDAR, using only RGB cameras is significantly harder as the robot must infer distances and geometry from flat images.

Methods

The system builds upon the Dreamer architecture, a model-based RL framework. The core components include:

  • Latent World Model: Instead of processing raw pixels for every decision, the system uses a Recurrent State Space Model (RSSM) to compress visual and sensor data into a "latent space." The robot then "imagines" future trajectories within this compressed space to plan actions.
  • Sensor Fusion: The model fuses segmented RGB images with non-visual data, including gait velocity, head position, and IMU data (pitch and roll). This provides the agent with "self-awareness" regarding its stability.
  • Termination Likelihood Predictor: A key contribution of this paper is an auxiliary loss function that predicts the likelihood of an episode ending in success (reaching the goal) or failure (falling). This helps the agent anticipate and avoid terminal states early in the learning process.
  • Sim-to-Real Transfer: To bridge the gap between simulation and reality, the system employs semantic segmentation. By converting raw camera feeds into simplified, color-coded maps of "obstacle vs. floor," the robot ignores real-world textures or lighting changes that were absent in the simulator.
The proposed architecture showing the interaction between the latent world model, policy, and termination predictors during training and inference.

Key Results and Insights

The model was trained entirely in the MuJoCo physics engine using eight parallel simulated environments. Evaluation on 100 random simulated scenes showed an 86% success rate.

Ablation studies highlighted the importance of specific components:

  • Predicting Failure: Without the termination predictor, the success rate dropped to 70%, proving that explicitly modeling failure improves sample efficiency.
  • The Necessity of Proprioception: Without non-visual (IMU/gait) data, the success rate plummeted to 0%. For a humanoid, internal stability data is as critical as external visual data.
  • Abstraction via Segmentation: Using segmented images allowed the robot to focus on task geometry rather than visual noise, facilitating a smooth sim-to-real transfer without retraining on real hardware.
The NimbRo-OP2X robot navigating a real-world environment using the learned policy and semantic segmentation overlays.

Conclusion and Future Work

The system successfully enabled a real robot to navigate narrow passages and avoid moving obstacles in real-time (10 Hz inference). However, limitations remain. The agent can occasionally get stuck in local minima during complex "back-up" maneuvers and lacks a high-level hierarchical planner for long-distance navigation. Future work will focus on incorporating hierarchical policies and estimating the velocity of dynamic obstacles to allow for more proactive path planning.

Summary updated

Abstract

In this paper, we propose a novel Deep Reinforcement Learning approach to address the mapless navigation problem, in which the locomotion actions of a humanoid robot are taken online based on the knowledge encoded in learned models. Planning happens by generating open-loop trajectories in a learned latent space that captures the dynamics of the environment. Our planner considers visual (RGB images) and non-visual observations (e.g., attitude estimations). This confers the agent upon awareness not only of the scenario, but also of its own state. In addition, we incorporate a termination likelihood predictor model as an auxiliary loss function of the control policy, which enables the agent to anticipate terminal states of success and failure. In this manner, the sample efficiency of the approach for episodic tasks is increased. Our model is evaluated on the NimbRo-OP2X humanoid robot that navigates in scenes avoiding collisions efficiently in simulation and with the real hardware.
BibTeX
@inproceedings{brandenburger2021mapless,
  author = {André Brandenburger and Diego Rodriguez and Sven Behnke},
  title = {{Mapless Humanoid Navigation Using Learned Latent Dynamics}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=164Z2g08_js},
}

Loading related content...

Figure