Reinforcement Learning with Prior Policy Guidance for Motion Planning of Dual-Arm Free-Floating Space Robot

Yuxue Cao, Shengjie Wang, Xiang Zheng, Wenke Ma, Xinru Xie, Lei LiuarXiv 2022 · OpenReview

TL;DR

This paper introduces EfficientLPT, a novel reinforcement learning (RL) approach for motion planning of dual-arm free-floating space robots. By combining a prior policy derived from inverse kinematics with a carefully designed reward function, EfficientLPT efficiently tackles the complex task of capturing rotating objects, overcoming limitations of previous RL methods in this domain.

What Problem Does This Paper Address?

Space robots are increasingly vital for on-orbit services, such as satellite repair. A key challenge is enabling dual-arm free-floating space robots to precisely manipulate objects in space. These robots are complex due to the dynamic coupling between the base satellite and the arms, and the high dimensionality of planning for two arms simultaneously. Existing RL methods struggle with this complexity, particularly when needing to capture non-cooperative objects that might be rotating. This is often due to a lack of pose constraints for the end-effectors and inefficient exploration in the vast state space, leading to slow convergence and difficulty in handling dynamic targets.

Methods

The core of this work is the EfficientLPT algorithm, which enhances standard RL techniques with two key innovations:

  • Mixed Policy with Prior Knowledge Guidance: Instead of relying solely on RL to learn from scratch (which can be slow and inefficient, especially in high-dimensional problems), EfficientLPT incorporates prior knowledge. Specifically, it uses the inverse kinematics of a fixed-base manipulator to generate a "prior policy." This prior policy provides a reasonable initial direction for the RL agent's exploration, guiding it towards optimal solutions more quickly. The final action is a weighted combination of the RL agent's output and this prior policy. This approach helps overcome the "cold-start" problem common in RL.
  • Efficient Reward Function: The paper proposes a novel reward function designed to accelerate convergence. For orientation error, it focuses on the maximum axis angle error (using an infinite norm) rather than averaging or summing errors across axes. This is because orientation is a larger state space and harder to explore. By penalizing the worst orientation error, the agent is more effectively guided towards accurate pose control. The reward also includes terms for position error and manipulability optimization.

The algorithm uses the Soft Actor-Critic (SAC) framework as its RL backbone, known for its sample efficiency and ability to handle continuous action spaces. The actions output by the RL agent are desired joint angular velocities, which are then converted into torques by a low-level PD controller.

Datasets

Not discussed in the paper. The experiments were conducted in a simulated environment.

Key Insights and Ideas

  • Prior knowledge is crucial for complex RL tasks: Directly applying RL to high-dimensional problems like dual-arm space robot manipulation can be inefficient. Injecting prior knowledge (like inverse kinematics) significantly speeds up learning and improves sample efficiency.
  • Focusing on the worst-case error can be more effective: For orientation control, penalizing the maximum axis angle error (infinite norm) is more effective than using L1, L2, or average cosine norms, leading to faster convergence and better performance.
  • Dual-arm manipulation in space is a challenging but solvable problem: The paper demonstrates that with the right approach, RL can successfully handle the complexities of free-floating dual-arm systems, including capturing rotating targets.

Key Results

  • High Success Rate: EfficientLPT achieved a 100% success rate in reaching stationary targets in the test environment after training, significantly outperforming pure RL (0%) and other reward function variants.
  • Effective Rotation Capture: The method successfully captured rotating targets with speeds up to 1.72°/s. While performance degraded at higher speeds (3.44°/s), it still met position error requirements, indicating robustness.
  • Improved Convergence: The mixed policy approach led to significantly more stable and faster convergence compared to a pure RL policy, especially in the early stages of training.
  • Robustness to Disturbances: The algorithm demonstrated robustness to base satellite motion disturbances, maintaining low position and orientation errors even with varying base masses.

Limitations and Future Work

  • Limitations with High-Speed Rotation: The current method struggles to meet orientation error requirements for targets rotating at higher speeds (e.g., 3.44°/s). This is partly due to strict limits on joint angular velocities to maintain stability.
  • Future Work: The authors suggest extending the method to handle tumbling targets with a wider range of motion in position and orientation. They also mention that the current approach is suitable for capturing slowly rotating targets, and for faster-rotating targets, a de-tumbling step might be necessary before capture. The paper also notes that the current work focuses on planning and does not involve force control after contact.

Summary updated

Abstract

Reinforcement learning methods as a promising technique have achieved superior results in the motion planning of free-floating space robots. However, due to the increase in planning dimension and the intensification of system dynamics coupling, the motion planning of dual-arm free-floating space robots remains an open challenge. In particular, the current study cannot handle the task of capturing a non-cooperative object due to the lack of the pose constraint of the end-effectors. To address the problem, we propose a novel algorithm, EfficientLPT, to facilitate RL-based methods to improve planning accuracy efficiently. Our core contributions are constructing a mixed policy with prior knowledge guidance and introducing infinite norm to build a more reasonable reward function. Furthermore, our method successfully captures a rotating object with different spinning speeds.
BibTeX
@misc{cao2022reinforcement,
  author = {Yuxue Cao and Shengjie Wang and Xiang Zheng and Wenke Ma and Xinru Xie and Lei Liu},
  title = {{Reinforcement Learning with Prior Policy Guidance for Motion Planning of Dual-Arm Free-Floating Space Robot}},
  year = {2022},
  eprint = {2209.01434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.01434},
}
Figure