Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking With a Wheeled-Quadrupedal-Manipulator

Kaiwen Jiang, Zhen Fu, Junde Guo, Wei Zhang, Hua Chen

TL;DR

This paper introduces a novel Reinforcement Learning (RL) approach for wheeled-quadrupedal robots to perform complex "loco-manipulation" tasks, where the robot must simultaneously move its base and manipulate objects with its arm. They achieve state-of-the-art 6D end-effector pose tracking by developing a "Reward Fusion Module" (RFM) that intelligently balances locomotion and manipulation objectives.

What Problem Does This Paper Address?

Robots that can both move around and interact with their environment (loco-manipulation) are crucial for many real-world applications. However, coordinating the complex movements of a robot's base (like legs and wheels) with its manipulator arm to precisely control the end-effector's position and orientation in 3D space is incredibly challenging. Existing methods often struggle with this full 6D pose tracking, sometimes requiring separate controllers for locomotion and manipulation, or only focusing on 3D position. This paper tackles the problem of achieving seamless, whole-body control for direct 6D end-effector pose tracking on a wheeled-quadrupedal robot, which inherently involves balancing many redundant degrees of freedom.

Methods

The core innovation is the Reward Fusion Module (RFM). Instead of simply summing up different reward signals (a linear approach), RFM uses a nonlinear fusion strategy to prioritize and balance rewards for various sub-tasks. This module is integrated into a teacher-student RL training paradigm.

Key components of the RFM include:

  • Reward Prioritization (RP): This mechanism establishes a hierarchy between rewards, for example, prioritizing position tracking over orientation tracking when the target is far away, and only focusing on orientation when the target is within reach. This is achieved through multiplicative relationships between reward terms.
  • Enhancement Module: This part focuses on improving tracking accuracy. It uses "micro-enhancement" to boost the reward gradient for small tracking errors and a "cumulative-based penalty mechanism" that emulates integral control to help the robot overcome local minima and eliminate steady-state errors.
  • Loco-Mani Fusion: This component smoothly transitions between locomotion and manipulation phases. It uses a phase variable that depends on the robot's distance to the target to dynamically shift focus between moving and manipulating, ensuring a fluid transition rather than abrupt switches.

The overall control pipeline uses a unified RL policy that takes proprioceptive states, a 6D target EE command, and privileged latent information (estimated from history) as input, outputting whole-body actions.

Datasets

The paper does not explicitly mention specific public datasets for training or evaluation. Instead, they train their RL policy in simulation using Isaac Gym and Mujoco, and then validate it on a real-world wheeled-quadrupedal robot platform. They also conduct ablation studies in simulation to quantitatively assess the effectiveness of their proposed RFM components.

Key Insights and Ideas

  • Nonlinear reward fusion is key for hierarchical tasks: Simply summing rewards doesn't capture the complex, multi-stage nature of loco-manipulation. A nonlinear approach allows for intelligent prioritization and balancing of objectives.
  • Task hierarchy matters for efficiency: Prioritizing locomotion when the target is far and manipulation when it's close is crucial for smooth and efficient task execution, preventing unnecessary arm movements during transit.
  • Cumulative error penalty aids in escaping local minima: Similar to integral control in traditional robotics, this mechanism helps the RL agent overcome challenging states and achieve precise final poses.
  • Smooth transitions are essential for real-world performance: Abrupt switches between locomotion and manipulation lead to jerky movements. The proposed fusion method enables fluid transitions, improving overall performance.

Key Results

  • State-of-the-art 6D EE pose tracking: The proposed method achieves tracking position errors of less than 5 cm and rotation errors of less than 0.1 radians.
  • Improved success rate and reduced errors: Ablation studies show that the RFM significantly outperforms baselines (e.g., standard PPO without RFM, or RFM without specific components) in terms of success rate, average power consumption, average acceleration, and tracking errors. For instance, the "w/o RFM" baseline had a significantly higher position error (0.168 m) compared to the proposed method (0.022 m).
  • Smooth loco-manipulation demonstrated: Hardware experiments showcase the robot seamlessly transitioning between locomotion and manipulation phases, even in dynamic scenarios like picking up objects.

Limitations and Future Work

  • Perceptive information: The authors acknowledge that their current end-to-end method lacks explicit perceptive input. They suggest that equipping the agent with perceptive capabilities would be critical for handling more complex scenarios, especially in confined spaces or when obstacles are present.
  • Generalization to more complex tasks: While impressive, the current work focuses on 6D EE pose tracking. Future work could explore more diverse and complex manipulation tasks.
  • Sim-to-real gap: Although they use techniques like domain randomization and a teacher-student approach, bridging the sim-to-real gap remains an ongoing challenge in RL for robotics.

Summary updated

Abstract

In this paper, we study the whole-body loco-manipulation problem using reinforcement learning (RL). Specifically, we focus on the problem of how to coordinate the floating base and the robotic arm of a wheeled-quadrupedal manipulator robot to achieve direct six-dimensional (6D) end-effector (EE) pose tracking in task space. Different from conventional whole-body loco-manipulation problems that track both floating-base and end-effector commands, the direct EE pose tracking problem requires inherent balance among redundant degrees of freedom in the whole-body motion. We leverage RL to solve this challenging problem. To address the associated difficulties, we develop a novel reward fusion module (RFM) that systematically integrates reward terms corresponding to different tasks in a nonlinear manner. In such a way, the inherent multi-stage and hierarchical feature of the loco-manipulation problem can be carefully accommodated. By combining the proposed RFM with the a teacher-student RL training paradigm, we present a complete RL scheme to achieve 6D EE pose tracking for the wheeled-quadruped manipulator robot. Extensive simulation and hardware experiments demonstrate the significance of the RFM. In particular, we enable smooth and precise tracking performance, achieving state-of-the-art tracking position error of less than 5 cm, and rotation error of less than 0.1 rd.
BibTeX
@article{jiang2024learning,
  author = {Kaiwen Jiang and Zhen Fu and Junde Guo and Wei Zhang and Hua Chen},
  title = {{Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking With a Wheeled-Quadrupedal-Manipulator}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2024},
  doi = {10.1109/lra.2024.3519856},
}
Figure