MIRACLE: Inverse Reinforcement and Curriculum Learning Model for Human-inspired Mobile Robot Navigation

Nihal Gunukula, Kshitij Tiwari, Aniket BeraarXiv 2023 · DOI · arXiv

Introduction

In emergency response scenarios, robots must navigate complex, partially-observable environments where traditional mathematical models often fail. Humans, conversely, are adept at interpreting subtle environmental stimuli—such as sounds or light—to locate goals. The MIRACLE (Inverse Reinforcement and Curriculum Learning Model) framework aims to bridge this gap by teaching mobile robots to navigate with human-like intuition through a combination of gamified Virtual Reality (VR) data collection and advanced machine learning.

Methodology

The MIRACLE framework integrates three primary components to translate human decision-making into robotic control policies.

The MIRACLE system architecture, illustrating the flow from human stimuli response in VR to robot navigation.

DeFINE: Gamified Data Collection

To overcome data scarcity, the researchers developed DeFINE, a Unity3D-based VR environment. Participants were tasked with finding a hidden goal guided by a "fly" stimulus. To ensure high-quality, optimized trajectories, the platform utilized a leaderboard to foster competition among participants. The study utilized 225 trajectories from 15 diverse individuals, recorded at high resolution (0.1-second intervals).

Deep Maximum Entropy Inverse Reinforcement Learning (IRL)

Rather than manually defining a reward function, the system uses IRL to infer the underlying rewards from human demonstrations. The "Maximum Entropy" approach allows the model to learn a distribution of paths, providing robustness against demonstrator bias and enabling the robot to handle environmental variability rather than blindly mimicking flawed human paths.

Reverse Curriculum Learning

A novel training strategy was employed where the model was first exposed to "peak performance" data (later trials from experienced participants) before being introduced to noisier, early-stage data. This grounded the model in expert behavior, ensuring more stable convergence and preventing the model from being misled by initial human learning errors.

Experimental Results

The framework was evaluated both in simulation and on physical hardware.

Overview of the MIRACLE pipeline: from human stimuli response learning in VR to autonomous robot navigation.

Simulation and Convergence

In a simulated environment, the MIRACLE model achieved a convergence loss of 2.7717. This low loss indicates a high degree of alignment between the learned policy and successful human navigational strategies, suggesting the model successfully captured the intended reward structure.

Real-World Deployment

The learned policy was successfully transferred to a Unitree GO1 quadruped robot. In physical trials, the robot demonstrated the ability to interpret a light source as a navigational stimulus, moving toward the goal autonomously. This confirms that the "intuition" captured in the VR environment is transferable to real-world robotic platforms.

Conclusion and Future Work

MIRACLE demonstrates a viable path for training robots in complex tasks using human-in-the-loop simulation. While successful, future research will focus on increasing environmental complexity (e.g., cluttered spaces), incorporating social navigation constraints (e.g., moving around people), and scaling the system to handle a wider variety of emergency stimuli.

Summary updated

Abstract

In emergency scenarios, mobile robots must navigate like humans, interpreting stimuli to locate potential victims rapidly without interfering with first responders. Existing socially-aware navigation algorithms face computational and adaptability challenges. To overcome these, we propose a solution, MIRACLE -- an inverse reinforcement and curriculum learning model, that employs gamified learning to gather stimuli-driven human navigational data. This data is then used to train a Deep Inverse Maximum Entropy Reinforcement Learning model, reducing reliance on demonstrator abilities. Testing reveals a low loss of 2.7717 within a 400-sized environment, signifying human-like response replication. Current databases lack comprehensive stimuli-driven data, necessitating our approach. By doing so, we enable robots to navigate emergency situations with human-like perception, enhancing their life-saving capabilities.
BibTeX
@misc{gunukula2023miracle,
  author = {Nihal Gunukula and Kshitij Tiwari and Aniket Bera},
  title = {{MIRACLE: Inverse Reinforcement and Curriculum Learning Model for Human-inspired Mobile Robot Navigation}},
  year = {2023},
  eprint = {2312.03651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.03651},
}
Figure