ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation

Fei Xia, Chengshu Li, Roberto Martín-Martín, Or Litany, Alexander Toshev, Silvio Savarese

TL;DR

This paper introduces ReLMoGen, a novel framework that integrates Reinforcement Learning (RL) with classical motion generation techniques for mobile manipulation tasks. By lifting the RL action space from low-level motor commands to high-level subgoals, ReLMoGen enables more efficient learning and successful execution of complex, long-horizon tasks that were previously intractable for pure RL methods.

What Problem Does This Paper Address?

Traditional Reinforcement Learning (RL) approaches for robotics often struggle with complex, long-horizon tasks like mobile manipulation. These tasks require a robot to navigate to specific locations and perform intricate interactions with its environment, often involving a long sequence of precise actions. The challenge lies in the vast state-action space and the difficulty of exploration, especially when rewards are sparse or delayed. Existing RL methods, which typically operate on low-level joint control signals (like velocities or torques), are inefficient for these tasks, leading to slow learning and often suboptimal solutions. While hierarchical RL offers a potential solution, it still requires learning a low-level policy from scratch, which can be data-intensive and unstable.

Methods

ReLMoGen tackles this problem by creating a "lifted" Markov Decision Process (MDP). Instead of directly learning a policy that outputs low-level motor commands, ReLMoGen learns a Subgoal Generation Policy (SGP) that outputs high-level subgoals. These subgoals are then passed to a pre-existing Motion Generator (MG), which is responsible for planning and executing collision-free, low-level trajectories to reach the specified subgoals.

The MG acts as a learned motion planner and trajectory executor, leveraging classical techniques like RRT-Connect. This separation of concerns allows the RL agent (SGP) to focus on the strategic decision-making of where to go or what to interact with, while the MG handles the low-level control of how to get there. ReLMoGen offers two instantiations of the SGP:

  • SGP-R (Regression): A continuous action space where the policy directly regresses base and arm subgoals.
  • SGP-D (Dense Prediction): A discrete action space where the policy outputs Q-value maps, similar to "spatial action maps," for base and arm subgoals.

This approach significantly reduces the effective action space for the RL agent, making exploration more efficient and learning more stable.

Datasets

The paper evaluates ReLMoGen on seven distinct tasks within the iGibson Environment [45], a photo-realistic simulation environment. These tasks span:

  • Navigation-Only: PointNav (reaching a goal location).
  • Manipulation-Only: TabletopReachM (reaching a point on a table with the end-effector).
  • Interactive Navigation (IN): PushDoorNav, ButtonDoorNav, InteractiveObstaclesNav (tasks requiring interaction with the environment to proceed).
  • Mobile Manipulation (MM): ArrangeKitchenMM, ArrangeChairMM (tasks involving both navigation and manipulation of objects).

The paper does not introduce new benchmarks but utilizes existing simulated environments and tasks to demonstrate the effectiveness of ReLMoGen.

Key Insights and Ideas

  • Lifting the Action Space is Key: The core idea is that for complex robotics tasks, learning to predict high-level subgoals is more tractable than learning low-level motor control directly. This temporal abstraction significantly aids exploration.
  • Synergy of Learning and Classical Methods: ReLMoGen effectively combines the strengths of deep RL (for strategic decision-making) and classical motion planning (for precise, collision-free execution). This hybrid approach overcomes the limitations of each individual method.
  • Motion Generators as "Skill Primitives": The MG acts as a robust, pre-defined "skill" that the RL agent can reliably call upon. This avoids the need for the RL agent to learn these low-level skills from scratch, which is a major bottleneck in traditional HRL.
  • Robustness to Motion Planners: The learned SGP demonstrates remarkable transferability, performing well even when paired with different motion planners (e.g., RRT-Connect and Lazy PRM) than those used during training. This suggests the subgoals are largely independent of the specific planning algorithm.
  • Interpretable Subgoals: The SGP-D's Q-value maps provide interpretable insights into what the agent considers important for task completion, often highlighting relevant objects or interaction points.

Key Results

  • Superior Performance: ReLMoGen consistently outperforms state-of-the-art RL (SAC, OAC) and Hierarchical RL (HRL4IN) baselines across all seven evaluated tasks, including challenging interactive navigation and mobile manipulation scenarios.
  • Higher Sample Efficiency: ReLMoGen achieves higher rewards with fewer environment episodes compared to baselines, indicating more efficient learning.
  • Faster Training: The framework leads to significantly shorter wall-clock training times (up to 7x faster) due to fewer gradient updates and more efficient exploration.
  • Generalization: ReLMoGen shows strong generalization capabilities, performing well with different motion planners and even demonstrating potential for sim-to-real transfer with minimal fine-tuning. For instance, on the PushDoorNav task, fine-tuning on a new robot embodiment (Movo) with only 2x10^4 episodes led to significant success rate improvements.

Limitations and Future Work

  • Reliance on Motion Generator: The success of ReLMoGen is contingent on the availability and effectiveness of a suitable motion generator for the given tasks. If the MG fails to find a plan for a subgoal, it returns a no-op, which can hinder learning.
  • Domain Gap: While the paper addresses the domain gap between simulation and the real world, it acknowledges that differences in perception and dynamics can still pose challenges. Future work could involve more advanced domain adaptation techniques.
  • Task Specificity: While ReLMoGen is general, the specific subgoals and interactions are learned. Future work could explore learning more generalizable interaction primitives or adapting to entirely new interaction types.
  • Fine-tuning for New Embodiments: While promising, zero-shot transfer to significantly different robot embodiments did not work well. Further research into more robust sim-to-real transfer and embodiment adaptation is suggested.

Summary updated

Abstract

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen – a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art RL and Hierarchical RL baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots. For more information, please visit project website: http://svl.stanford.edu/projects/relmogen.
BibTeX
@inproceedings{xia2021relmogen,
  author = {Fei Xia and Chengshu Li and Roberto Martín-Martín and Or Litany and Alexander Toshev and Silvio Savarese},
  title = {{ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561315},
}

Loading related content...

Figure