Eye and Voice-Controlled Human Machine Interface System for Wheelchairs Using Image Gradient Approach

Saba Anwer, Asim Waris, Hajrah Sultan, Shahid Ikramullah Butt, Muhammad Hamza Zafar, Moaz Sarwar, Imran Khan Niazi, Muhammad Shafique, Amit N. PujariSensors 2020 · DOI

TL;DR

This paper presents a novel, low-cost eye and voice-controlled human-machine interface (HMI) for wheelchairs, designed to enhance mobility and independence for individuals with severe physical disabilities. By leveraging image gradient techniques for eye tracking and integrating voice commands, the system offers a more accessible and user-friendly alternative to existing, often cumbersome, HMI solutions.

What Problem Does This Paper Address?

Conventional wheelchairs, while beneficial, often fall short for individuals with severe mobility impairments like quadriplegia or paralysis. These users struggle to operate standard controls, necessitating advanced assistive technologies. Existing human-machine interfaces (HMIs) for such users, while improving, often rely on complex biosignals (EEG, EMG) that can be uncomfortable, prone to noise, or require wearable equipment. Other eye-tracking systems can be bulky, expensive, or even pose risks to eye health (e.g., infrared illumination). The challenge lies in developing an HMI that is precise, reliable, comfortable, scalable, and affordable, enabling greater independence for those with limited motor function.

Methods

The system employs a two-pronged approach: eye-tracking and voice control.

  • Eye-Tracking: This core component utilizes a webcam to capture real-time images of the user's eyes. The system then applies an image gradient approach to locate the eye center. This method defines the eye center as the intersection of multiple image gradients. Specifically, it calculates gradient vectors and uses a mathematical formula to determine the center of a circular object (the eyeball) by minimizing the dot product between displacement vectors and gradient vectors. This technique is chosen for its relative simplicity and effectiveness in low-resolution images, avoiding the need for head-mounted or wearable sensors.
  • Voice Control: For an alternative control method, an adaptive multi-rate AMR voice app is integrated. This allows users to issue commands (forward, left, right, stop) via voice.
  • Hardware Integration: A Raspberry Pi serves as the central processing unit, running an Open CV library for image processing and eye-tracking algorithms. An Arduino UNO acts as the main controller for both eye and voice commands, interfacing with a four-channel relay module to drive the wheelchair's DC motors. A Bluetooth module (HC-05) facilitates voice command transmission.

Datasets

Not discussed in the paper. The system is evaluated using a custom "Basic Skills Performance Test" (WST) conducted with 15 test subjects.

Key Insights and Ideas

  • Image Gradients for Eye Tracking: The use of image gradients for eye center localization is a key innovation, offering a practical and less intrusive alternative to other eye-tracking methods. It avoids the need for specialized hardware attached to the user.
  • Dual Control Modality: The integration of both eye and voice control significantly enhances usability and adaptability. This provides users with flexibility, especially if one mode becomes difficult to use in certain situations.
  • Low-Cost and Compact Design: The reliance on a Raspberry Pi and readily available components makes the system significantly more affordable and compact than many existing solutions, increasing its accessibility.
  • Focus on User Comfort and Safety: The design prioritizes user comfort by carefully positioning the camera and avoiding potentially harmful technologies like active infrared illumination.

Key Results

The system demonstrated successful operation with an average response time of 3 seconds for eye control and 3.4 seconds for voice control. The Basic Skills Performance Test (WST) involved 15 participants performing various maneuvers (forward, backward, turn in place, turn while moving, rolling over soft surface) over a 25m distance.

  • Response Times:
    • Forward move: Eye = 2.53s, Voice = 2.8s
    • Backward move: Eye = 3.4s, Voice = 3.0s
    • Turn in place: Eye = 2.86s, Voice = 2.9s
    • Turn while moving: Eye = 2.78s, Voice = 2.89s
    • Rolling over soft surface: Eye = 3.46s, Voice = 3.5s
  • Performance: All participants successfully completed the 25m distance without intervention. The system showed high responsiveness with 99% of participants reporting no false-positive results.

Limitations and Future Work

  • Illumination Sensitivity: The paper acknowledges that image processing techniques can sometimes malfunction in low-light conditions due to variations in illumination. While a 12V LED is incorporated to mitigate this, further improvements are needed.
  • Future Work: The authors suggest exploring FPGA-field programmable gate array systems to enhance processing speed and improve synchronization with environmental variations and user needs. This could lead to even faster response times and greater robustness.

Summary updated

BibTeX
@article{anwer2020eye,
  author = {Saba Anwer and Asim Waris and Hajrah Sultan and Shahid Ikramullah Butt and Muhammad Hamza Zafar and Moaz Sarwar and Imran Khan Niazi and Muhammad Shafique and Amit N. Pujari},
  title = {{Eye and Voice-Controlled Human Machine Interface System for Wheelchairs Using Image Gradient Approach}},
  journal = {Sensors},
  year = {2020},
  doi = {10.3390/s20195510},
}
Figure