A soft artificial muscle driven robot with reinforcement learning

Tao Yang, Youhua Xiao, Zhen Zhang, Yiming Liang, Guorui Li, Mingqi Zhang, Shijian Li, Tuck‐Whye Wong, Yong Wang, Tiefeng Li, Zhilong HuangScientific 2018 · DOI

TL;DR

This paper introduces a novel soft robot that mimics a cuttlefish, powered entirely by artificial muscles made from dielectric elastomers. By employing reinforcement learning to optimize control, they significantly boosted the robot's swimming speed, demonstrating a promising approach for controlling complex soft robotic systems.

What Problem Does This Paper Address?

Traditional robots, with their rigid components, offer high precision and control but struggle in unstructured or delicate environments. Soft robots, inspired by nature, offer advantages like adaptability, resilience, and inherent safety. However, their very flexibility and the complex, non-linear nature of their stimuli-responsive actuators (like dielectric elastomers) make precise control a significant challenge. This paper tackles this control problem, specifically for a soft underwater robot, aiming to achieve efficient and high-performance locomotion without relying on traditional motors.

Methods

The core of the robot's actuation comes from dielectric elastomer (DE) membranes. These materials deform significantly when a high voltage is applied, mimicking muscle action. The researchers designed a "jet-actuator" where DE membranes, sandwiched with carbon grease and electrodes, are integrated into an acrylic chamber. When voltage is applied, the DE membrane expands, drawing water into the chamber. Upon discharge, the water is expelled through an orifice, generating thrust. To enhance this propulsion, they incorporated a magnetic biasing mechanism that amplifies the DE's displacement.

The key innovation lies in the control strategy. Instead of manual tuning, they employed Reinforcement Learning (RL), specifically Q-learning with experience replay. This approach allows the robot to learn an optimal control policy through trial and error. The robot's state is defined by its recent actions (voltage inputs), and its actions are the discrete voltage levels (0 kV or 6.8 kV) applied to the DE actuators. The reward function is designed to maximize the robot's displacement, thereby increasing its swimming speed. This RL algorithm iteratively updates a state-action value table to learn the best sequence of voltage inputs for efficient locomotion.

Datasets

Not discussed in the paper. The paper focuses on a single robot system and its control strategy, rather than training on pre-existing datasets.

Key Insights and Ideas

  • DEs as powerful, yet challenging, actuators: Dielectric elastomers offer impressive actuation capabilities (large deformation, high energy density) but their inherent non-linearity makes them difficult to control precisely with conventional methods.
  • Biomimicry for propulsion: Mimicking the jet propulsion of cephalopods (like cuttlefish) provides an effective mechanism for soft underwater robots.
  • Reinforcement Learning for soft robot control: RL is a powerful paradigm for overcoming the control challenges posed by soft robots, enabling them to learn complex behaviors autonomously.
  • Integrated system design: The successful implementation requires not just the actuator but also a compact, onboard power supply and wireless communication system, demonstrating a holistic approach to soft robot design.
  • Magnetic biasing enhances DE performance: The use of magnets significantly amplifies the displacement of the DE actuators, leading to more effective propulsion.

Key Results

The paper reports a significant improvement in the cuttlefish robot's swimming performance after applying the reinforcement learning control strategy.

  • Swimming Speed Enhancement: The robot's swimming speed was increased by 91%, reaching a maximum of 21 mm/s (or 0.38 body lengths per second). This is a substantial improvement compared to the initial performance without RL.
  • Optimized Actuation Pattern: The RL process led to a convergence of the driving voltage sequence into a more periodic and optimized pattern, indicating a learned control strategy.
  • Successful Untethered Operation: The robot was successfully operated untethered, demonstrating the viability of the integrated onboard system.

Limitations and Future Work

  • Directional Control: The current robot can only move forward and lacks directional control. The authors suggest that incorporating additional soft actuators could enable steering.
  • Hydrodynamic Complexity: The RL process's fluctuations are attributed to the complexity of the RL itself, actuation motion, and hydrodynamic drag. Further research could refine the state representation to better account for these factors.
  • Material Efficiency: The FEA simulations revealed inhomogeneous deformation in the DE membrane, suggesting potential for material optimization through variable thickness designs.
  • Generalizability: While successful for this cuttlefish robot, the RL strategy's applicability to other soft robot designs and tasks would require further investigation.

Summary updated

Abstract

Soft robots driven by stimuli-responsive materials have their own unique advantages over traditional rigid robots such as large actuation, light weight, good flexibility and biocompatibility. However, the large actuation of soft robots inherently co-exists with difficulty in control with high precision. This article presents a soft artificial muscle driven robot mimicking cuttlefish with a fully integrated on-board system including power supply and wireless communication system. Without any motors, the movements of the cuttlefish robot are solely actuated by dielectric elastomer which exhibits muscle-like properties including large deformation and high energy density. Reinforcement learning is used to optimize the control strategy of the cuttlefish robot instead of manual adjustment. From scratch, the swimming speed of the robot is enhanced by 91% with reinforcement learning, reaching to 21 mm/s (0.38 body length per second). The design principle behind the structure and the control of the robot can be potentially useful in guiding device designs for demanding applications such as flexible devices and soft robots.
BibTeX
@inproceedings{yang2018soft,
  author = {Tao Yang and Youhua Xiao and Zhen Zhang and Yiming Liang and Guorui Li and Mingqi Zhang and Shijian Li and Tuck‐Whye Wong and Yong Wang and Tiefeng Li and Zhilong Huang},
  title = {{A soft artificial muscle driven robot with reinforcement learning}},
  booktitle = {Scientific},
  year = {2018},
  doi = {10.1038/s41598-018-32757-9},
}
Figure