A Novel Gesture-Based Language for Underwater Human–Robot Interaction

Davide Chiarella, Marco Bibuli, Gabriele Bruzzone, Massimo Caccia, Andrea Ranieri, Enrica Zereik, Lucía Marconi, Paola CutugnoJournal of Marine Science and Engineering 2018 · DOI

TL;DR

This paper introduces "Caddian," a novel gesture-based language designed for intuitive communication between divers and underwater robots. By leveraging existing diving hand signals and formalizing them into a robust protocol, Caddian aims to enhance safety and cooperation in challenging underwater environments.

What Problem Does This Paper Address?

Underwater operations are inherently risky due to poor visibility, communication limitations, and the difficulty of monitoring divers. Traditional communication methods are often unreliable or impossible. This paper tackles the critical need for a natural, effective, and robust communication system between divers and their robotic companions (like Autonomous Underwater Vehicles - AUVs) to improve safety, efficiency, and collaboration during dives. Existing underwater communication methods, such as acoustics, suffer from low data rates and high costs, while radio and optical methods are limited by the environment.

Methods

The core of this work is the development of the "Caddian" language. This language is built upon:

  • Consolidated Gestures: It extends established and universally accepted gestures used by divers, aiming for backward compatibility and easier adoption.
  • Formal Language Definition: Caddian is defined using formal grammar principles (Backus-Naur Form - BNF), providing a structured syntax and semantics.
  • Bijective Mapping: Gestures are mapped to a written alphabet (Latin letters) and vice-versa, allowing for a clear translation between physical actions and symbolic representation.
  • Semantic Function: Sequences of gestures are translated into commands or messages for the robot.
  • Communication Protocol: A protocol is established to ensure error handling, including mechanisms for the robot to confirm understanding and for divers to correct or abort commands. This protocol also defines how the robot communicates its status (idle, busy, failure) using lights.
  • Gesture Recognition: While the paper focuses on the language design, it acknowledges the underlying challenge of robust gesture recognition in the harsh underwater environment, mentioning techniques like skin detection, motion analysis, and stereocamera systems.

Datasets

Not discussed in the paper. The paper focuses on the design and testing of the language itself, not on training a gesture recognition model from scratch on a large dataset. The "datasets" used for evaluation were the trials themselves, involving divers performing gestures.

Key Insights and Ideas

  • Leveraging Existing Knowledge: Building upon existing diver gestures significantly lowers the learning curve and increases the likelihood of adoption, making the system more practical.
  • Formalization is Key: Defining Caddian with a formal grammar provides a structured and unambiguous way for robots to interpret diver commands, crucial for reliable operation.
  • Error Handling is Paramount: The communication protocol's emphasis on error detection and correction (e.g., robot confirmation, diver feedback) is vital for safety and trust in a high-stakes environment.
  • Underwater Challenges Drive Design: The inherent difficulties of underwater communication (visibility, bubbles, illumination) directly influenced the choice of gestures and the need for a robust, non-acoustic system.
  • Intuition Over Complexity: The decision to avoid dynamic gestures in the initial trials, despite their potential richness, highlights a pragmatic approach to balancing functionality with the limitations of current underwater computer vision.

Key Results

  • Successful Field Trials: The paper reports on experimental campaigns at sea and in a pool, demonstrating the feasibility of the Caddian language.
  • High Recognition Rates for Simple Gestures: For single gestures, the system achieved good recognition capabilities, even in challenging underwater conditions (low visibility, currents, waves).
  • Encouraging Complex Sequence Recognition: Complex gesture sequences (dialogues/sentences) showed success rates ranging from 87% to 99.9%, with the "do a mosaic" mission being an exception at 60%.
  • Rapid Language Learning: On dry land tests, volunteers learned the language (22 gestures) very quickly, with error rates decreasing significantly after only a few missions, indicating an acceptable cognitive load.
  • Robustness in Varied Conditions: The gesture recognition performed well even in "stormy weather" and low visibility conditions (2.5m with suspended sediments).

Limitations and Future Work

  • Limited Gesture Set in Trials: The trials used a minimal subset of the language (22 gestures), and dynamic gestures were not included due to recognition challenges.
  • Natural Number Syntax: The syntax for natural numbers was found to be less intuitive than initially designed.
  • "Do a Mosaic" Mission: This specific complex mission had a lower success rate (60%), suggesting areas for improvement.
  • Future Work:
    • Testing the full language set, including dynamic gestures.
  • Further improving the success rate for certain gestures and refining the natural number syntax.
  • Adding the ability to change missions with updated parameters or create parameterized missions (macros/functions).
  • Conducting deeper studies on the whole language in new scenarios, potentially creating new commands.
  • Collecting more data to prove the robustness of the CADDY framework.
  • Gathering diver feedback to further refine and adapt the language.

Summary updated

Abstract

The underwater environment is characterized by hazardous conditions that make it difficult to manage and monitor even the simplest human operation. The introduction of a robot companion with the task of supporting and monitoring the divers during their activities and operations underwater can help to solve some of the problems that usually arise in this scenario. In this context, a proper communication between the diver and the robot is imperative for the success of the dive. However, the underwater environment poses a set of technical challenges which are not readily surmountable thus limiting the spectrum from which possibilities can be chosen. This paper presents the design and development of a gesture-based communication language which has been employed for the entire duration of the European project CADDY (Cognitive Autonomous Diving Buddy). This language, the Caddian, was built upon consolidated and standardized underwater gestures that are commonly used in recreational and professional diving. Its use and integration during field tests with a remotely operated underwater vehicle (ROV) is also shown.
BibTeX
@article{chiarella2018novel,
  author = {Davide Chiarella and Marco Bibuli and Gabriele Bruzzone and Massimo Caccia and Andrea Ranieri and Enrica Zereik and Lucía Marconi and Paola Cutugno},
  title = {{A Novel Gesture-Based Language for Underwater Human–Robot Interaction}},
  journal = {Journal of Marine Science and Engineering},
  year = {2018},
  doi = {10.3390/jmse6030091},
}
Figure