Spatial Action Maps for Mobile Manipulation

Jimmy Wu, Xingyuan Sun, Andy Zeng, Shuran Song, Johnny Lee, Szymon Rusinkiewicz, Thomas A. Funkhouser

TL;DR

This paper introduces "spatial action maps," a novel dense action representation for mobile manipulation tasks. By aligning action predictions with the visual state space, their method significantly improves sample efficiency and performance compared to traditional discrete steering commands, enabling robots to learn complex behaviors like pushing objects into a receptacle with far fewer training examples.

What Problem Does This Paper Address?

Robotic manipulation, especially in dynamic and unseen environments, is a challenging problem. Traditional approaches to training robots for tasks like navigation and object manipulation often rely on predicting discrete, low-level steering commands (e.g., "turn left," "move forward"). This presents several issues: these actions are myopic, only allowing for small changes in state, requiring long sequences of actions to achieve significant progress. This also necessitates complex mappings from high-dimensional visual inputs to these low-dimensional actions, demanding vast amounts of training data. The authors aim to bridge this gap by proposing a more intuitive and efficient action representation that aligns with the robot's visual perception.

Methods

The core contribution is the "spatial action map" representation. Instead of discrete commands, the robot predicts a dense map where each pixel corresponds to a local navigational endpoint in the robot's local coordinate frame. This map is spatially aligned with the input bird's-eye view (BEV) of the environment, which is constructed using an Inverse Perspective Mapping (IPM) technique.

The system uses a fully convolutional neural network (FCN), specifically a ResNet-18 backbone, to map the input state (a 4-channel image including the overhead map, robot position, shortest path distance from the agent, and shortest path distance to the receptacle) to the spatial action map. This map represents the predicted Q-values for navigating to each potential endpoint. The robot then selects the endpoint with the highest Q-value and executes a movement primitive to reach it. Two movement primitives are explored: a straight-line path and a shortest path (using an occupancy map).

The learning is framed as a reinforcement learning problem, specifically using a Double DQN formulation. Rewards are given for successfully pushing objects into a receptacle, with partial rewards based on changes in shortest path distance to the receptacle, and penalties for collisions or lack of movement.

Datasets

The paper primarily uses a simulated environment (PyBullet) for training and evaluation. They create four types of environments with increasing difficulty: SmallEmpty, SmallColumns, LargeColumns, and LargeDivider. These environments feature randomized object and obstacle placements. The trained policies are then tested on a real Anki Vector robot in a simplified version of the SmallEmpty environment. No new datasets are introduced; rather, the focus is on evaluating the action representation within existing simulation frameworks.

Key Insights and Ideas

  • Spatial Alignment is Key: Aligning the action space (spatial action maps) with the state space (BEV images) allows for a more direct and efficient mapping using FCNs. This pixel-to-pixel correspondence simplifies learning.
  • Dense Actions for Efficiency: Representing actions as local endpoints rather than discrete commands allows for much larger state changes per action. This drastically reduces the number of actions needed to achieve goals, leading to significantly improved sample efficiency.
  • Shortest Path Matters: Incorporating shortest path information (both as input channels and in movement primitives) is crucial for effective navigation, especially in environments with obstacles. It helps the agent plan more intelligently and avoid getting stuck.
  • Partial Rewards are Beneficial: Using partial rewards based on shortest path distance to the receptacle, rather than just Euclidean distance, provides a stronger learning signal, particularly in complex environments.
  • Emergent Behaviors: The learned policies exhibit interesting emergent behaviors, such as using walls as guides for sweeping multiple objects, demonstrating the system's ability to learn complex strategies.

Key Results

  • Superior Sample Efficiency: The proposed spatial action map approach achieved significantly better performance and sample efficiency compared to a baseline using traditional steering commands. For instance, in the LargeDivider environment, the spatial action map agent was much more effective, while the steering command baseline struggled to learn.
  • High Performance in Complex Tasks: The agent successfully learned to push objects into a receptacle in various environments, achieving high success rates (e.g., 18.23 objects pushed in LargeDivider).
  • Generalization to Real World: Policies trained in simulation transferred directly to a physical Anki Vector robot, demonstrating the robustness of the approach. The real robot could push 8.4 out of 10 objects within 15 minutes in a simplified environment.
  • Ablation Studies Confirm Value: Removing shortest path components (input channels, movement primitive, or partial rewards) led to performance degradation, especially in more challenging environments, highlighting the importance of these features.

Limitations and Future Work

  • Simulation-to-Real Gap: The sim-to-real transfer is limited by the accuracy of the simulation, physical properties, and the robot's localization and mapping capabilities. Fine-tuning in the physical setup could address some of these issues.
  • Motion Primitive Assumption: Spatial action maps rely on high-level motion primitives to handle lower-level control. This assumes that these primitives can be handled separately, and end-to-end learning of both navigation and manipulation might be more beneficial for certain tasks.
  • Action Representation Scope: The current action representation is limited to navigating to a spatial location and performing a discrete action. Extending this to higher-dimensional settings or tasks that inherently mix navigation and manipulation is a future direction.
  • Future Work: The authors suggest exploring other action spaces that blend navigation and manipulation, as well as applying the approach to different application domains like autonomous driving. Investigating how to incorporate uncertainty in state and observations into planning is also an active research area.

Summary updated

Abstract

Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e.g., step forward, turn left, turn right, etc.) from images of the current state (e.g., a bird's-eye view of a SLAM reconstruction). Instead, we show that it can be advantageous to learn with dense action representations defined in the same domain as the state. In this work, we present "spatial action maps," in which the set of possible actions is represented by a pixel map (aligned with the input image of the current state), where each pixel represents a local navigational endpoint at the corresponding scene location. Using ConvNets to infer spatial action maps from state images, action predictions are thereby spatially anchored on local visual features in the scene, enabling significantly faster learning of complex behaviors for mobile manipulation tasks with reinforcement learning. In our experiments, we task a robot with pushing objects to a goal location, and find that policies learned with spatial action maps achieve much better performance than traditional alternatives.
BibTeX
@inproceedings{wu2020spatial,
  author = {Jimmy Wu and Xingyuan Sun and Andy Zeng and Shuran Song and Johnny Lee and Szymon Rusinkiewicz and Thomas A. Funkhouser},
  title = {{Spatial Action Maps for Mobile Manipulation}},
  booktitle = {Robotics: Science and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=zfQZjGu4bAoc},
}
Figure