TL;DR
This paper introduces EfficientLPT, a novel reinforcement learning (RL) approach for motion planning of dual-arm free-floating space robots. By combining a prior policy derived from inverse kinematics with a carefully designed reward function, EfficientLPT efficiently tackles the complex task of capturing rotating objects, overcoming limitations of previous RL methods in this domain.
What Problem Does This Paper Address?
Space robots are increasingly vital for on-orbit services, such as satellite repair. A key challenge is enabling dual-arm free-floating space robots to precisely manipulate objects in space. These robots are complex due to the dynamic coupling between the base satellite and the arms, and the high dimensionality of planning for two arms simultaneously. Existing RL methods struggle with this complexity, particularly when needing to capture non-cooperative objects that might be rotating. This is often due to a lack of pose constraints for the end-effectors and inefficient exploration in the vast state space, leading to slow convergence and difficulty in handling dynamic targets.
Methods
The core of this work is the EfficientLPT algorithm, which enhances standard RL techniques with two key innovations:
- Mixed Policy with Prior Knowledge Guidance: Instead of relying solely on RL to learn from scratch (which can be slow and inefficient, especially in high-dimensional problems), EfficientLPT incorporates prior knowledge. Specifically, it uses the inverse kinematics of a fixed-base manipulator to generate a "prior policy." This prior policy provides a reasonable initial direction for the RL agent's exploration, guiding it towards optimal solutions more quickly. The final action is a weighted combination of the RL agent's output and this prior policy. This approach helps overcome the "cold-start" problem common in RL.
- Efficient Reward Function: The paper proposes a novel reward function designed to accelerate convergence. For orientation error, it focuses on the maximum axis angle error (using an infinite norm) rather than averaging or summing errors across axes. This is because orientation is a larger state space and harder to explore. By penalizing the worst orientation error, the agent is more effectively guided towards accurate pose control. The reward also includes terms for position error and manipulability optimization.
The algorithm uses the Soft Actor-Critic (SAC) framework as its RL backbone, known for its sample efficiency and ability to handle continuous action spaces. The actions output by the RL agent are desired joint angular velocities, which are then converted into torques by a low-level PD controller.
Datasets
Not discussed in the paper. The experiments were conducted in a simulated environment.
Key Insights and Ideas
- Prior knowledge is crucial for complex RL tasks: Directly applying RL to high-dimensional problems like dual-arm space robot manipulation can be inefficient. Injecting prior knowledge (like inverse kinematics) significantly speeds up learning and improves sample efficiency.
- Focusing on the worst-case error can be more effective: For orientation control, penalizing the maximum axis angle error (infinite norm) is more effective than using L1, L2, or average cosine norms, leading to faster convergence and better performance.
- Dual-arm manipulation in space is a challenging but solvable problem: The paper demonstrates that with the right approach, RL can successfully handle the complexities of free-floating dual-arm systems, including capturing rotating targets.
Key Results
- High Success Rate: EfficientLPT achieved a 100% success rate in reaching stationary targets in the test environment after training, significantly outperforming pure RL (0%) and other reward function variants.
- Effective Rotation Capture: The method successfully captured rotating targets with speeds up to 1.72°/s. While performance degraded at higher speeds (3.44°/s), it still met position error requirements, indicating robustness.
- Improved Convergence: The mixed policy approach led to significantly more stable and faster convergence compared to a pure RL policy, especially in the early stages of training.
- Robustness to Disturbances: The algorithm demonstrated robustness to base satellite motion disturbances, maintaining low position and orientation errors even with varying base masses.
Limitations and Future Work
- Limitations with High-Speed Rotation: The current method struggles to meet orientation error requirements for targets rotating at higher speeds (e.g., 3.44°/s). This is partly due to strict limits on joint angular velocities to maintain stability.
- Future Work: The authors suggest extending the method to handle tumbling targets with a wider range of motion in position and orientation. They also mention that the current approach is suitable for capturing slowly rotating targets, and for faster-rotating targets, a de-tumbling step might be necessary before capture. The paper also notes that the current work focuses on planning and does not involve force control after contact.
Summary updated