Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces

Wenbo Liu, Youning Duo, Jiaqi Liu, Feiyang Yuan, Lei Li, Luchen Li, Gang Wang, Bohan Chen, Siqi Wang, Hui Yang, Yu‐Chen Liu, Yanru Mo, Yun Wang, Bin Fang, Fuchun Sun, Xilun Ding, Chi Zhang, Li WenNature Communications 2022 · DOI

TL;DR

This paper introduces a novel "flexible bimodal smart skin" (FBSS) that allows humans to teach soft robots complex movements through touchless hand gestures. This system enables non-experts to intuitively program soft robots for various tasks, paving the way for broader human-robot collaboration.

What Problem Does This Paper Address?

Soft robots offer significant advantages in human-robot interaction due to their inherent safety and conformability. However, their complex modeling and programming present a major hurdle for non-specialists, limiting their widespread adoption in real-world applications. Traditional "contact teaching" methods used for rigid robots are not suitable for soft robots due to their infinite degrees of freedom and compliant nature. This paper addresses the need for an intuitive, non-programming-based method to teach soft robots complex movements and tasks, bridging the gap between human intent and robotic execution.

Methods

The core innovation is the Flexible Bimodal Smart Skin (FBSS), a multi-layered sensor that combines two sensing modalities:

  • Triboelectric Nanogenerator (TENG): This layer is responsible for touchless sensing. It generates electrical signals based on the electrostatic induction caused by the proximity of an external object.
  • Liquid Metal Sensor: This layer handles tactile sensing. It detects physical contact by measuring changes in the resistance of embedded liquid metal channels, which are compressed or deformed upon contact.

A key advancement is the FBSS's ability to simultaneously and distinguishably sense both tactile and touchless stimuli in real-time. This is achieved by leveraging the distinct physical principles of each sensing mechanism.

For teaching, the researchers developed a "shifting sensors and teaching" method. This involves:

  1. Touchless Control: Users interact with the FBSS by moving their hands at varying distances. The distance of the hand from the FBSS dictates the magnitude of the touchless signal, which is then translated into specific robot movements (e.g., bending, shortening).
  2. Shifting the Sensor: The FBSS can be physically moved to different segments of the soft manipulator. This allows users to selectively teach movements to specific parts of the robot.
  3. Tactile Confirmation: Pressing the FBSS acts as a trigger to "lock" a learned movement or initiate a task like grasping.

The system then uses a kinematic model to translate the sensor signals into pneumatic actuation commands for the soft manipulator.

Datasets

Not discussed in the paper. The paper focuses on demonstrating the capabilities of the FBSS and the teaching method through various experimental tasks rather than training a machine learning model on a specific dataset.

Key Insights and Ideas

  • Unambiguous Tactile vs. Touchless Sensing: The ability of a single sensor to reliably differentiate between proximity (touchless) and contact (tactile) is a significant breakthrough. This overcomes a major challenge in previous triboelectric-based sensors.
  • Intuitive Human-Robot Interaction: The touchless, hand-eye coordination approach makes teaching natural and accessible to non-experts, eliminating the need for complex programming or specialized robotics knowledge.
  • "Shifting Sensors and Teaching" for Complex Locomotion: This method elegantly addresses the challenge of controlling multi-segment soft robots by allowing users to focus on teaching individual segments sequentially, enabling complex 2D and 3D movements.
  • Broad Applicability: The FBSS and teaching method are demonstrated on various tasks, including maze navigation, throat swabbing, and object grasping, highlighting their versatility.

Key Results

  • Effective Teaching: Participants could effectively teach complex 3D motions to a soft continuum manipulator in just a few minutes.
  • Motion Replay: The taught motions could be accurately replayed by the soft manipulator at different speeds.
  • Task Completion: The soft manipulator successfully completed tasks such as:
  • Completing a pen-and-paper maze.
  • Performing a simulated throat swab.
  • Crossing a barrier to grasp an object.
  • Grasping objects at various heights.
  • Non-Specialist Usability: Novice users demonstrated the ability to learn and accurately position the soft manipulator's end effector with minimal error after a few attempts.
  • Robustness: The FBSS showed good stability and durability over 1200 loading-unloading cycles and was relatively unaffected by environmental factors like temperature, humidity, and electromagnetic interference.

Limitations and Future Work

  • Limited Number of Sensors: The current study uses two FBSS units for 3D control. The authors suggest incorporating more interactive soft sensors to enable even more complex controls and feedback.
  • Machine Learning Integration: While the current system relies on direct mapping of sensor signals to robot actions, the authors envision using machine learning with multi-sensor arrays to recognize human gestures and environmental objects for more sophisticated interactions.
  • Object/Obstacle Distinction: The ability to distinguish between a human hand, a target object, and an obstacle in complex environments is an area for future development.
  • Material Limitations: While the FBSS can sense a wide range of materials, the touchless sensing mechanism relies on electrostatic induction, which might be influenced by the dielectric properties of the objects.
  • Future Applications: The authors foresee broader applications in collaborative robotics, healthcare, and other domains where intuitive human-robot interaction is crucial.

Summary updated

Abstract

In this paper, we propose a multimodal flexible sensory interface for interactively teaching soft robots to perform skilled locomotion using bare human hands. First, we develop a flexible bimodal smart skin (FBSS) based on triboelectric nanogenerator and liquid metal sensing that can perform simultaneous tactile and touchless sensing and distinguish these two modes in real time. With the FBSS, soft robots can react on their own to tactile and touchless stimuli. We then propose a distance control method that enabled humans to teach soft robots movements via bare hand-eye coordination. The results showed that participants can effectively teach a self-reacting soft continuum manipulator complex motions in three-dimensional space through a "shifting sensors and teaching" method within just a few minutes. The soft manipulator can repeat the human-taught motions and replay them at different speeds. Finally, we demonstrate that humans can easily teach the soft manipulator to complete specific tasks such as completing a pen-and-paper maze, taking a throat swab, and crossing a barrier to grasp an object. We envision that this user-friendly, non-programmable teaching method based on flexible multimodal sensory interfaces could broadly expand the domains in which humans interact with and utilize soft robots.
BibTeX
@inproceedings{liu2022touchless,
  author = {Wenbo Liu and Youning Duo and Jiaqi Liu and Feiyang Yuan and Lei Li and Luchen Li and Gang Wang and Bohan Chen and Siqi Wang and Hui Yang and Yu‐Chen Liu and Yanru Mo and Yun Wang and Bin Fang and Fuchun Sun and Xilun Ding and Chi Zhang and Li Wen},
  title = {{Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces}},
  booktitle = {Nature Communications},
  year = {2022},
  doi = {10.1038/s41467-022-32702-5},
}
Figure