Spatial Intention Maps for Multi-Agent Mobile Manipulation

Jimmy Wu, Xingyuan Sun, Andy Zeng, Shuran Song, Szymon Rusinkiewicz, Thomas A. Funkhouser

TL;DR

This paper introduces "spatial intention maps," a novel way for decentralized multi-agent robots to communicate their intentions by rendering them into a 2D map aligned with their visual observations. This representation significantly improves coordination and cooperative behaviors in complex physical tasks, leading to better performance and emergent strategies.

What Problem Does This Paper Address?

Coordinating multiple robots in shared physical spaces to perform collaborative tasks is a significant challenge, especially when robots operate decentrally and have partial observability. Existing methods for communicating intentions often rely on high-level state information or low-dimensional embeddings, which fail to capture crucial spatial relationships. This makes it difficult for deep reinforcement learning (RL) agents, particularly those using convolutional neural networks (CNNs) that excel at processing visual data, to effectively leverage intention information for spatial coordination. The paper aims to bridge this gap by proposing a spatially explicit intention representation that integrates seamlessly with vision-based RL.

Methods

The core contribution is the spatial intention map. Instead of simply sharing coordinates or abstract representations, each agent's intention (represented as its intended path) is rendered into a 2D map. This map is pixel-wise aligned with the agent's state map (derived from its visual observations) and its action map.

Here's a breakdown of the key methodological aspects:

  • Intention Representation: An agent's intended path is encoded as a linear ramp function within the spatial intention map. The value is 1 at the agent's current location and decreases linearly along the path, indicating the time until it reaches that point. This provides a richer signal than just a destination coordinate.
  • Integration with Deep RL: The spatial intention map is fed as an input channel to a fully convolutional deep residual Q-network (similar to the Spatial Action Maps framework [8]). This allows the network to process intention information in the same spatial domain as the visual state, enabling it to learn emergent cooperative behaviors.
  • Decentralized Execution: While trained using a centralized approach (sharing policies within robot types), the agents execute actions decentrally and asynchronously. This means each robot makes decisions based on its local observations and the communicated intentions of others.
  • Communication Mechanism: To keep bandwidth low, agents broadcast their intentions as lists of (x, y) coordinates. Each agent then locally renders these intentions into its spatial intention map.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated environments for training and evaluation. These environments are designed to test various aspects of multi-agent coordination, including:

  • Foraging Task: Robots must move objects to a target receptacle.
  • Search and Rescue Task: Robots must locate and "rescue" objects.

The environments vary in size (Small/Large) and complexity (Empty, Divider, Doors, Tunnels, Rooms), incorporating obstacles and requiring coordination through narrow passages. The trained policies are also tested on real robots in a sim-to-real setup.

Key Insights and Ideas

  • Spatial Encoding is Crucial: The paper strongly suggests that representing intentions spatially is far more effective than non-spatial encodings for vision-based multi-agent RL. This allows the CNNs to naturally process and reason about intentions alongside visual input.
  • Emergent Cooperative Behaviors: Spatial intention maps enable robots to learn sophisticated cooperative strategies without explicit programming. Examples include:
  • Collision Avoidance: Robots proactively adjust their paths to avoid each other.
  • Coordinated Passage: Robots learn to navigate through bottlenecks (like doorways) by implicitly coordinating their movements.
  • Task Specialization (Heterogeneous Teams): In teams with robots of different abilities (lifting, pushing, throwing), spatial intention maps help facilitate a natural division of labor, leading to more efficient task completion.
  • Reduced Conservative Behavior: Without intention maps, robots tend to move conservatively to minimize collisions. With spatial intention maps, they can take more direct actions, knowing how others will behave.
  • Sim-to-Real Transfer: Policies trained in simulation with spatial intention maps demonstrate successful generalization to real-world robotic platforms.

Key Results

  • Improved Foraging Performance: In various environments, teams using spatial intention maps consistently outperformed teams without them. For instance, lifting robot teams gathered an average of over 19 objects in large environments with intention maps, compared to only 11-17 without.
  • Enhanced Search and Rescue Efficiency: Spatial intention maps led to more efficient rescue operations, with robots avoiding overlapping efforts and achieving better distribution.
  • Effective Heterogeneous Teams: Spatial intention maps significantly improved the performance of heterogeneous teams, preventing some robots from wandering aimlessly and ensuring more productive collaboration.
  • Qualitative Improvements: Visualizations of Q-value maps and robot trajectories demonstrate how intention maps guide robots to make better decisions, such as choosing unoccupied doorways or rescuing different objects to avoid redundancy.

Limitations and Future Work

  • Communication Bandwidth: While the paper aims for low bandwidth by broadcasting waypoints, the rendering process is done locally. The paper notes that "predicted intention" methods (training a separate network to predict intention maps) can achieve performance close to explicit communication with spatial intention maps, suggesting a path towards even more efficient communication.
  • Computational Cost: Training deep RL models can be computationally intensive. The paper mentions that predicting intention maps adds extra computation.
  • Generalization to Unknown Environments: While sim-to-real transfer was demonstrated, further work could explore generalization to entirely unseen environments or task variations.
  • More Complex Interactions: Future work could explore more complex multi-agent scenarios, such as mixed cooperative-competitive settings or tasks requiring more intricate forms of communication and coordination. The paper also suggests exploring more sophisticated intention representations beyond simple paths.

Summary updated

Abstract

The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks. In this work, we present spatial intention maps, a new intention representation for multi-agent vision-based deep reinforcement learning that improves coordination between decentralized mobile manipulators. In this representation, each agent’s intention is provided to other agents, and rendered into an overhead 2D map aligned with visual observations. This synergizes with the recently proposed spatial action maps framework, in which state and action representations are spatially aligned, providing inductive biases that encourage emergent cooperative behaviors requiring spatial coordination, such as passing objects to each other or avoiding collisions. Experiments across a variety of multi-agent environments, including heterogeneous robot teams with different abilities (lifting, pushing, or throwing), show that incorporating spatial intention maps improves performance for different mobile manipulation tasks while significantly enhancing cooperative behaviors.
BibTeX
@inproceedings{wu2021spatial,
  author = {Jimmy Wu and Xingyuan Sun and Andy Zeng and Shuran Song and Szymon Rusinkiewicz and Thomas A. Funkhouser},
  title = {{Spatial Intention Maps for Multi-Agent Mobile Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=zKLawTo-09q},
}
Figure