A Human-Machine Interface Based on an EOG and a Gyroscope for Humanoid Robot Control and Its Application to Home Services

Fan Wang, Xiongzi Li, Jiahui PanJournal of Healthcare Engineering 2022 · DOI

TL;DR

This paper introduces a novel asynchronous human-machine interface (HMI) for controlling humanoid robots, combining electrooculography (EOG) for double-blink detection and a gyroscope for head rotation. This system allows users to issue a rich set of commands with high accuracy and low latency, demonstrating improved performance in simulated home service tasks compared to traditional manual control.

What Problem Does This Paper Address?

The core challenge addressed is the development of effective human-machine interfaces (HMIs) for robot teleoperation, particularly for individuals with motor disabilities. Existing manual interfaces like joysticks and keyboards become unusable for those with severe neuromuscular disorders. While non-manual HMIs like Brain-Computer Interfaces (BCIs) using EEG or EOG show promise, they often struggle with the trade-off between providing a sufficient number of commands and maintaining high accuracy with low response times. Specifically, traditional BCI paradigms like P300 and SSVEP are limited by the number of targets they can present without increasing response time. EOG-based systems, while offering good signal-to-noise ratios, have faced challenges in creating asynchronous systems that provide ample commands with low latency, often relying on synchronous flashing cues or limited eye movements.

Methods

The proposed HMI utilizes two key inputs:

  • Electrooculogram (EOG): Specifically, a "double blink" gesture is used as a trigger. This is detected by filtering the EOG signal, identifying two distinct peaks (peak1 and peak2), and analyzing their duration and energy.
  • Gyroscope: This device measures head rotation, providing yaw and roll angles. These angles are used to calculate cursor movement on a graphical user interface (GUI).

These inputs are integrated into a multilevel Graphical User Interface (GUI). The GUI features a navigation panel and a joint control panel.

  • Navigation Panel: Offers 8 navigation buttons and 3 speed control buttons.
  • Joint Control Panel: Operates in three levels:
  • Joint Selection: Users choose a specific joint (e.g., head, shoulder, elbow).
  • Joint Motion Selection: Based on the selected joint, users choose from 2-4 motion options (e.g., two-axis rotations).
  • Stopping Level: A large button to halt any ongoing motion.

A novel continuous control strategy is implemented. This strategy breaks down robot actions into two phases: a "motion start" phase initiated by selecting a navigation or joint command, and a "motion stop" phase triggered by the double blink when the robot reaches the desired state. This avoids the need to combine multiple commands for a single action or convert complex motions into discrete commands.

The system's algorithm involves:

  1. Cursor Position Calculation: Using gyroscope yaw and roll angles to update cursor position on the GUI.
  2. Double Blink Detection: Employing Pearson correlation coefficient (PCC) to assess similarity to a double-blink segment and extracting features like duration and energy to confirm a blink.
  3. Threshold Calculation: A calibration process is performed for each user to determine personalized thresholds for blink detection.

Datasets

Not discussed in the paper. The experiments were conducted with human subjects performing tasks in a simulated environment.

Key Insights and Ideas

  • Asynchronous Control with Rich Commands: The HMI successfully achieves an asynchronous control paradigm, allowing users to issue commands at their own pace, while simultaneously providing a comprehensive set of 40 commands (navigation, joint motion, speed, and stop).
  • Double Blink as a Robust Trigger: The double blink, combined with cursor position, proves to be a reliable and accurate trigger mechanism, achieving very high accuracy in button-clicking tasks.
  • Continuous Control Strategy: Disassembling robot motion into "start" and "stop" phases significantly simplifies control and leads to more natural, fluid robot movements, reducing the cognitive load on the user.
  • Synergy of EOG and Gyroscope: The combination of eye movements (for triggering) and head movements (for cursor control) offers a powerful and intuitive way to interact with the robot.
  • Improved Home Service Task Performance: The system demonstrates a tangible benefit in practical applications, showing improved efficiency and command usage in simulated home service scenarios compared to manual control.

Key Results

  • Target Button Triggering (Experiment I):
    • Accuracy (ACC): 99.2%
    • Response Time (RT): 2.34 seconds
    • False Positive Rate (FPR): 0.34 events/min (idle state)
    • The system outperformed previous BCI and EOG-based HMIs in terms of ACC and RT.
  • Humanoid Robot Control for Home Services (Experiment II):
    • Average Accuracy (ACC): 99.3%
    • Average Information Transfer Rate (ITR): 113.9 bits/min
    • Time Ratio (HMI vs. Manual Control): 1.22 (meaning HMI control took 1.22 times longer on average)
    • Command Ratio (HMI vs. Manual Control): 1.18 (meaning HMI control used 1.18 times more commands on average)
    • All subjects completed the home service tasks without failure.
  • The results indicate that the proposed HMI and continuous control strategy can improve performance in humanoid robot control for home services.

Limitations and Future Work

  • Visual Feedback: Subjects in the home service experiment required more visual feedback beyond the robot's monocular vision to accurately perceive positional relationships between the robot's arm and objects.
  • Machine Vision Comparison: The HMI control was not as precise as machine vision for object manipulation, as automatic control can leverage location data for optimal trajectory planning.
  • Future Work:
    • Reduce FPR: Improve the EOG detection algorithm and refine the continuous control strategy to further minimize false commands.
  • Shared Control System: Integrate machine vision guidance with HMI control to create a shared control system. This would allow HMI to handle navigation and object identification, while machine vision handles automatic manipulation, reducing complexity and increasing precision.
  • Automatic Grasp System: Develop an automatic grasp system guided by machine vision.

Summary updated

Abstract

The human-machine interface (HMI) has been studied for robot teleoperation with the aim of empowering people who experience motor disabilities to increase their interaction with the physical environment. The challenge of an HMI for robot control is to rapidly, accurately, and sufficiently produce control commands. In this paper, an asynchronous HMI based on an electrooculogram (EOG) and a gyroscope is proposed using two self-paced and endogenous features, double blink and head rotation. By designing the multilevel graphical user interface (GUI), the user can rotate his head to move the cursor of the GUI and create a double blink to trigger the button in the interface. The proposed HMI is able to supply sufficient commands at the same time with high accuracy (ACC) and low response time (RT). In the trigger task of sixteen healthy subjects, the target was clicked from 20 options with ACC of 99.2% and RT 2.34 s. Furthermore, a continuous strategy that uses motion start and motion stop commands to create a certain robot motion is proposed to control a humanoid robot based on the HMI. It avoids the situation that combines some commands to achieve one motion or converts the certain motion to a command directly. In the home service experiment, all subjects operated a humanoid robot changing the state of a switch, grasping a key, and putting it into a box. The time ratio between HMI control and manual control was 1.22, and the number of commands ratio was 1.18. The results demonstrated that the continuous strategy and proposed HMI can improve performance in humanoid robot control.
BibTeX
@article{wang2022human,
  author = {Fan Wang and Xiongzi Li and Jiahui Pan},
  title = {{A Human-Machine Interface Based on an EOG and a Gyroscope for Humanoid Robot Control and Its Application to Home Services}},
  journal = {Journal of Healthcare Engineering},
  year = {2022},
  doi = {10.1155/2022/1650387},
}
Figure