Policy learning in SE(3) action spaces

Dian Wang, Colin Kohler, Robert Platt

This paper, presented at CoRL 2020, introduces a framework for teaching robots to perform complex manipulation tasks—like building houses out of blocks—in full 6-degree-of-freedom (6-DOF) space. By cleverly factoring the action space and refining imitation learning techniques, the authors demonstrate that robots can learn to handle novel shapes and unstable environments (like ramps) far more effectively than with previous methods.

TL;DR

The authors propose ASRSE3, a method that breaks down high-dimensional robotic actions into a sequence of smaller steps, and SDQfD, an improved imitation learning algorithm. Together, these allow robots to learn complex 6-DOF manipulation policies that outperform standard baselines in both simulation and real-world block-building tasks.

What Problem Does This Paper Address?

Robotic manipulation usually relies on one of two action representations:

  1. Low-level joint control: Flexible but requires reasoning over hundreds of tiny steps, making reinforcement learning (RL) painfully slow.
  2. Spatial actions: The robot picks a target pose (e.g., "place the block at $x, y, z$ with rotation $\theta$"). This is much faster but is typically limited to 2D or 3D spaces (SE(2)).

Scaling spatial actions to the full 6-DOF of SE(3)—which includes position ($x, y, z$) and orientation (roll, pitch, yaw)—is difficult because the action space becomes massive. Standard Q-learning struggles to explore or accurately value such a huge number of possibilities. The authors aim to bridge this gap, enabling robots to reason about complex 3D orientations and heights without the computational explosion.

Methods

The paper introduces two core technical contributions:

1. ASRSE3 (Augmented State Representation for SE(3))

Instead of trying to predict a single 6D action vector all at once, ASRSE3 transforms the problem into a sequence of "partial actions." As shown in Figure 2, a single high-dimensional transition is broken into a chain of smaller decisions.

Figure 2: Augmented state representa- tion for A = A1 × A2 × A3.

In practice, the model uses a series of five Q-networks ($q_1$ through $q_5$) to select action components sequentially:

  • $q_1$: Picks the $(x, y)$ position using a U-Net on a top-down heightmap.
  • $q_2$ - $q_5$: Sequentially pick rotation ($\theta$), height ($z$), and out-of-plane tilts ($\phi, \psi$). Each subsequent network receives a "cropped" and "rotated" view of the scene based on the previous choices, as illustrated in Figure 3.
Figure 3: Our model selects actions a1 = axy, a2 = aθ, a3 = az, a4 = aφ, and a5 = aψ sequentially. q1 maps the heightmap image onto a |X| × |Y | map of Q values with a maximum at axy. Given that selection of axy, q2 predicts Q values for aθ. Similarly, q3, q4, and q5 predict Q values for az, aφ, and aψ, respectively. The input of in-hand image H and the output head se- lection accomplished by g is omitted in the figure. (a): the top-down heightmap I. (b-e) the output of f2-f5.

2. SDQfD (Strict Deep Q-learning from Demonstrations)

To guide exploration in these complex tasks, the authors use imitation learning. They improve upon the standard DQfD algorithm by introducing a "Strict" large margin loss. While the original DQfD only penalizes the single best non-expert action, SDQfD penalizes all non-expert actions that have a Q-value higher than the expert's action (minus a margin). This provides a much stronger supervisory signal in large action spaces where many sub-optimal actions might otherwise be over-estimated.

Datasets

The researchers evaluated their methods on a suite of 11 block construction tasks (Figure 4), ranging from simple two-block stacks to complex "houses" built with randomized shapes and heights.

Figure 4: Experimental environments.
  • Simulation: Used PyBullet for training and initial testing. They introduced "improvise" environments where block dimensions and shapes (Figure 16 in the paper) are randomized, forcing the agent to adapt to novel geometry.
  • Expert Data: Generated via "structure deconstruction"—an automated heuristic that takes a finished structure and records the steps to take it apart, then reverses them to create expert construction demonstrations.
  • Real-world: A UR5 robot arm with a Robotiq gripper and a depth sensor was used to validate that the learned policies transfer to physical hardware.

Key Insights and Ideas

  • Factoring is Efficient: By breaking the 6D action into a sequence, the model can use specialized architectures (like U-Nets for spatial reasoning) for some dimensions while using simpler convolutional crops for others. This is computationally much cheaper than a "brute force" 6D grid.
  • n-step Returns are Vital: In the augmented MDP, a single "real" robot move is now multiple "algorithmic" steps. The authors found that using $n$-step returns to propagate rewards from the final placement back to the initial $(x, y)$ choice was crucial for convergence.
  • Strictness Matters: In high-dimensional spaces, there are many ways to be "wrong." SDQfD's approach of suppressing all over-valued non-expert actions makes the imitation signal significantly more robust.

Key Results

  • Superior Success Rates: ASRSE3 DQN significantly outperformed standard FCN-DQN and DDPG baselines, especially on challenging tasks like the 4-block stack (4S), where precision is key to prevent toppling (Figure 5).
  • Imitation Boost: SDQfD outperformed the best versions of previous imitation learning methods (ADET and DQfD) by an average of 4% and 16%, respectively.
  • Real-World Transfer: The system successfully built complex structures on a real robot, even when the blocks were placed on inclined ramps, requiring the agent to utilize its 6-DOF reasoning to place blocks at an angle.
Comparison of ASRSE3 DQN against baselines.

Limitations and Future Work

  • Orientation Range: The current implementation only handles out-of-plane rotations ($\phi, \psi$) within a $\pm 30^\circ$ range. Full $360^\circ$ 3D orientation remains a challenge.
  • Expert Dependency: The method requires an expert planner (or the deconstruction heuristic) to provide initial demonstrations, which might not be available for all types of tasks.
  • Task Scope: While successful in block building, the authors note that extending this to more varied domestic manipulation tasks (like clearing a table or loading a dishwasher) is a critical next step.

Summary updated

Abstract

In the spatial action representation, the action space spans the space of target poses for robot motion commands, i.e. SE(2) or SE(3). This approach has been used to solve challenging robotic manipulation problems and shows promise. However, the method is often limited to a three dimensional action space and short horizon tasks. This paper proposes ASRSE3, a new method for handling higher dimensional spatial action spaces that transforms an original MDP with high dimensional action space into a new MDP with reduced action space and augmented state space. We also propose SDQfD, a variation of DQfD designed for large action spaces. ASRSE3 and SDQfD are evaluated in the context of a set of challenging block construction tasks. We show that both methods outperform standard baselines and can be used in practice on real robotics systems.
BibTeX
@inproceedings{wang2020policy,
  author = {Dian Wang and Colin Kohler and Robert Platt},
  title = {{Policy learning in SE(3) action spaces}},
  booktitle = {Conference on Robot Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=5h1-E0fLPm4},
}
Figure