Robot in a China Shop: Using Reinforcement Learning for Location-Specific Navigation Behaviour

Xihan Bian, Oscar Mendez, Simon HadfieldarXiv 2021 · arXiv

TL;DR

The MERLIN architecture treats visual navigation as a multi-task learning problem, allowing robots to adapt their behavior to specific environments—moving cautiously in a "china shop" while speeding through a hallway. By blending outputs from multiple "expert" networks and employing an auxiliary environment classification loss, the system achieves higher accuracy and a 26% faster convergence rate than current state-of-the-art models.

What Problem Does This Paper Address?

Most modern robotic navigation systems are designed to operate within a single class of environment. While a robot might excel at vacuuming a specific house, scaling that intelligence to navigate an entire city or a diverse set of indoor spaces (like moving from a cramped bathroom to a sprawling warehouse) remains a significant challenge.

The authors argue that robots need to mimic human behavior: we change how we move based on our surroundings, not just our task. In a crowded shop, we are slow and precise; in an empty corridor, we prioritize speed. Current AI models often suffer from "negative learning"—where learning a new environment causes the agent to forget how to navigate a previous one—or they fail to scale efficiently as the number of environments increases.

Methods

The authors propose a new architecture called MERLIN (Multi-environment Reinforcement-Learning in Navigation). The system is built on four primary components:

  1. Siamese Feature Extractor: This module takes two inputs—the robot's current view and an image of its target destination. It uses shared weights (based on a ResNet50 backbone) to extract features that are useful for both navigation and identifying the environment type.
  2. Task-Specific Expert Policy Sub-Networks: Instead of one giant network, MERLIN uses $n$ smaller "expert" networks. Each expert learns the specific nuances of a particular environment type.
  3. Attentive Task Allocation (Soft Blending): Rather than hard-switching between experts, an attention mechanism assigns weights to each expert’s output. These are then blended to produce the final navigation policy and value function. This "soft blending" allows the robot to leverage shared knowledge across similar environments.
  4. Environment Classifier (Room Classifier): This is a crucial auxiliary branch. It encourages the feature extractor to preserve information that helps identify the room type. Interestingly, this classifier is only used during training to accelerate the learning process.
Fig. 2: Our architecture can be divided into 4 major components: A Siamese feature extractor (green), n sub-expert-networks (blue), attention network (yellow), and a environment classifier network (orange).

The entire system is trained using the Asynchronous Advantage Actor-Critic (A3C) reinforcement learning algorithm.

Datasets

The researchers evaluated MERLIN in both simulation and the real world:

  • Simulated: They used AI2-THOR, a 3D environment providing themed rooms like kitchens, living rooms, and bathrooms.
  • Real-World: Data was collected using a Turtlebot in various locations, including university common rooms, residential living areas, and office corridors. The researchers created gridded versions of these real-world spaces to facilitate off-line training.

Key Insights and Ideas

  • Auxiliary Loss as a Catalyst: The inclusion of the Environment Classifier isn't just for labeling; it acts as a powerful regularizer. By forcing the network to recognize where it is early in the pipeline, the model learns navigation features much faster.
  • The Power of Shared Expertise: Surprisingly, the MERLIN model often outperformed specialist experts even on their own "home turf." This suggests that learning to navigate a kitchen actually provides transferable skills that help in navigating a living room, provided the architecture can manage that knowledge correctly.
  • Behavioral Adaptation: The model successfully learned location-specific traits. In narrow spaces like bathrooms, the agent prioritized moving away from walls. In open spaces, it took larger "strides" and focused more on turning for localization.
Qualitative examples of the agent's behavior. The green lines show trajectories, demonstrating how the agent adapts its pathing to the geometry of different rooms.

Key Results

  • Efficiency: MERLIN achieved a 24%–26% reduction in training time compared to jointly trained baseline models.
  • Accuracy: In simulations, MERLIN reached a 99.5% success rate, outperforming the "Joint Expert" baseline (99.3%) and significantly beating individual specialists who failed when placed in new environments.
  • Robustness: The model showed impressive noise resilience. It maintained over an 80% success rate even when the visual input was subjected to 50% noise, indicating strong generalization capabilities.

Limitations and Future Work

The primary limitation noted is the reliance on discrete actions (e.g., fixed 90-degree turns on a grid). In real-world deployments, robots face "drifting" errors where turns aren't perfect. While the model attempted to correct these by turning until it recognized a landmark, this strategy struggles with excessive lag or drift.

Future research directions include:

  • Developing methods for smooth transitions between expert networks as a robot moves between rooms.
  • Exploring lifelong learning where the robot can autonomously decompose complex new tasks into simpler ones handled by new independent expert networks.
  • Improving generalization to handle continuous movement and non-standard rotation angles.

Summary updated

Abstract

Robots need to be able to work in multiple different environments. Even when performing similar tasks, different behaviour should be deployed to best fit the current environment. In this paper, We propose a new approach to navigation, where it is treated as a multi-task learning problem. This enables the robot to learn to behave differently in visual navigation tasks for different environments while also learning shared expertise across environments. We evaluated our approach in both simulated environments as well as real-world data. Our method allows our system to converge with a 26% reduction in training time, while also increasing accuracy.
BibTeX
@misc{bian2021robot,
  author = {Xihan Bian and Oscar Mendez and Simon Hadfield},
  title = {{Robot in a China Shop: Using Reinforcement Learning for Location-Specific Navigation Behaviour}},
  year = {2021},
  eprint = {2106.01434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.01434},
}
Figure