Characterization of continuum robot arms under reinforcement learning and derived improvements

Ryota Morimoto, Masahiro Ikeda, Ryuma Niiyama, Yasuo KuniyoshiFrontiers Robotics AI 2022 · DOI

TL;DR

This paper investigates the performance of continuum robot arms in various tasks using reinforcement learning, comparing them to traditional rigid robots. They found that continuum arms excel in tasks with environmental uncertainty, while rigid arms are better at precise manipulation. The authors also propose a novel reinforcement learning method to improve continuum robot performance by regulating initial states.

What Problem Does This Paper Address?

Controlling soft continuum robot arms is challenging due to their inherent flexibility and infinite degrees of freedom, making traditional modeling difficult. While reinforcement learning (RL) offers a data-driven approach to overcome this, most studies focus on specific tasks with custom-built robots, lacking a comprehensive comparison with established rigid manipulators. This paper aims to bridge this gap by:

  1. Characterizing continuum robot arms: Understanding their strengths and weaknesses across a range of tasks compared to rigid counterparts.
  2. Evaluating RL performance: Investigating how RL performs on these different tasks for both robot types.
  3. Developing improved RL methods: Proposing a novel RL approach to enhance continuum robot control, specifically addressing challenges identified during the characterization.

The authors highlight that existing research often uses robots unique to each lab, limiting generalizability. They also point out that RL for soft robots is typically confined to simpler tasks like reaching, whereas this study explores more complex scenarios.

Methods

The core of this research utilizes Soft Actor-Critic (SAC), a state-of-the-art model-free, off-policy reinforcement learning algorithm. SAC is chosen for its ability to handle continuous state and action spaces and its maximum entropy objective, which promotes robustness and sample efficiency.

Key methodological aspects include:

  • Simulation Environment: A physics-based simulation environment (MuJoCo) was developed to mimic a real pneumatic continuum robot arm and a 7-DoF rigid robot arm. This allows for extensive experimentation without the cost and time constraints of real-world robotics.
  • Robot Models:
    • Continuum Robot Arm: Modeled as a series of interconnected segments with pneumatic actuators (EPABs), allowing for continuous bending and extension.
  • 7-DoF Rigid Robot Arm: A standard industrial robot arm (Fetch robot) was used for comparison, featuring discrete joints.
  • Task Design: Four distinct tasks were chosen to probe different robot capabilities:
  • Reaching: Basic position control.
  • Crank Rotation: Tasks involving interaction with an uncertain environment and cumulative rewards.
  • Peg-in-Hole: Tasks requiring precise position and force control.
  • Ball Throwing: Tasks assessing dynamic movement and anisotropy.
  • Randomization: To evaluate robustness, experiments incorporated randomness in the initial state of the robot and the environment (e.g., target positions, handle angles, hole locations).
  • Proposed Improvement: A novel "reset phase" was introduced. This involves allowing the continuum robot arm to contract to a neutral state for a short period before each episode begins. This aims to mitigate the negative impact of initial posture randomness, particularly in tasks requiring precise control.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it relies on data generated through extensive simulations of the defined tasks. The experiments themselves serve as the data generation mechanism for training and evaluating the RL agents.

Key Insights and Ideas

  • Task-Specific Strengths: Continuum robot arms excel in tasks with environmental uncertainty and where compliance is beneficial (like crank rotation), while rigid arms are superior in tasks demanding high precision (like peg-in-hole).
  • Anisotropy and Symmetry: Continuum arms, due to their symmetrical structure, are more robust to directional variations (anisotropy) in tasks like ball throwing, whereas rigid arms show more directional dependence.
  • Impact of Randomness: Initial state randomness significantly affects continuum robot arm learning, especially in precise tasks. Rigid robots are generally less sensitive to this.
  • Obstacles as a Challenge: While continuum arms can handle some environmental clutter, large obstacles can severely hinder their performance, especially when precise manipulation is needed.
  • The "Reset Phase" Hypothesis: The idea that pre-conditioning the continuum robot arm to a known, neutral state before an episode can significantly improve learning performance in precise tasks is a key contribution. This addresses the inherent challenge of their flexible nature.

Key Results

  • Crank Rotation: Continuum arms performed better in this task, especially when environmental randomness was present, suggesting their adaptability to uncertainty. Rigid arms showed slower learning with environmental randomness.
  • Peg-in-Hole: Rigid arms significantly outperformed continuum arms in this precision-demanding task. The continuum arm's performance was heavily degraded by initial posture randomness.
  • Ball Throwing: Continuum arms showed less directional dependence (anisotropy) compared to rigid arms, indicating better performance in tasks requiring consistent force application regardless of direction.
  • Simulation Validity: The study confirmed that their simulation model is a reliable substitute for real-world experiments, with similar learning trends observed between simulated and real continuum robot arms.
  • Reset Phase Effectiveness: The proposed "reset phase" method showed improvements in cumulative reward for continuum robot arms in peg-in-hole tasks, particularly when initial posture randomness was high. However, it did not significantly benefit the crank rotation task, where less precise control is needed.

Limitations and Future Work

  • Limited Task Scope: While four tasks were explored, the authors acknowledge that a wider variety of tasks would provide a more comprehensive understanding of continuum robot arm characteristics.
  • Anisotropy in Ball Throwing: While continuum arms were better, the rigid arm's performance varied significantly with direction, suggesting further investigation into anisotropy is needed.
  • Obstacle Influence: The impact of obstacles on continuum robot arms cannot be ignored, and future work should focus on developing methods to mitigate this influence while maintaining tolerance to environmental randomness.
  • Generalizability of RL Method: The proposed "reset phase" is a simple yet effective method, but further research could explore more sophisticated RL techniques to enhance continuum robot control.
  • Other Soft Robot Properties: The study focused on specific aspects. Future work could explore other properties like viscoelasticity and the influence of different actuator types.

Summary updated

BibTeX
@article{morimoto2022characterization,
  author = {Ryota Morimoto and Masahiro Ikeda and Ryuma Niiyama and Yasuo Kuniyoshi},
  title = {{Characterization of continuum robot arms under reinforcement learning and derived improvements}},
  journal = {Frontiers Robotics AI},
  year = {2022},
  doi = {10.3389/frobt.2022.895388},
}
Figure