Abstract
In this paper, we propose a novel touchless typing interface that makes use of an on-screen QWERTY keyboard and a smartphone camera. The keyboard was divided into nine color-coded clusters. The user moved their head toward clusters, which contained the letters that they wanted to type. A front-facing smartphone camera recorded the head movements. A bidirectional GRU based model which used pre-trained embedding rich in head pose features was employed to translate the recordings into cluster sequences. The model achieved an accuracy of 96.78% and 86.81% under intra- and inter-user scenarios, respectively, over a dataset of 2234 video sequences collected from 22 users.
Introduction
For individuals with physical conditions like quadriplegia, interacting with traditional peripherals like keyboards or mice is impossible. While assistive technologies exist, they often come with significant trade-offs. Speech recognition can fail for those with speaking disabilities or in noisy environments, while Brain-Computer Interfaces (BCI) are often expensive and intrusive. Eye-tracking, though effective, can be physically strenuous and requires specialized hardware.
This research introduces a system that is low-cost, non-intrusive, and intuitive. By leveraging a standard smartphone camera and the familiar QWERTY layout, the system allows users to "type" through head movements, providing a promising alternative for individuals with limited upper-body mobility.
System Architecture and Methods
The system pipeline consists of three main stages: interface design, feature extraction, and sequence modeling.
- Interface Design: The standard QWERTY keyboard is divided into nine color-coded clusters. Instead of targeting individual keys, users move their head toward the cluster containing their desired letter. This reduces the complexity from a 26-way classification problem to a 9-way one.
- Feature Extraction: A front-facing camera records the user, and the system uses HopeNet (a ResNet50-based model) to estimate head pose (Yaw, Pitch, and Roll) for each frame. These Euler angles are converted into a 198-dimensional feature embedding.
- Sequence Modeling: These embeddings are fed into a 20-layered Bidirectional Gated Recurrent Unit (Bi-GRU). The authors used Connectionist Temporal Classification (CTC) loss to handle the alignment between video frames and typed letters. Finally, Beam Search Decoding translates the model's output into the most likely cluster sequence, which is then mapped to actual words.
Experimental Setup and Dataset
The authors collected a new dataset involving 22 users (19 male, 3 female). Each user typed a variety of words, phrases, and sentences, resulting in 2,234 video sequences recorded at 30fps using a standard smartphone.
A key design choice was the QWERTY layout; because users are already familiar with it, the learning curve is significantly reduced. Analysis showed that for the 10,000 most common English words, a cluster sequence maps to only 1.17 different words on average, allowing the system to disambiguate user intent with high precision.
Key Results
The system demonstrated robust performance across different testing scenarios:
- Accuracy: The model achieved 96.78% accuracy in intra-user scenarios and 86.81% in inter-user (user-invariant) scenarios.
- Typing Speed: Users averaged 49.26 gestures per minute. While slower than manual typing, this is a competitive rate for touchless assistive interfaces.
- Error Analysis: Using a Modified Dynamic Time Warping (M-DTW) metric that considers spatial coordinates, the authors found that even when the model predicted the wrong cluster, it was usually a cluster physically adjacent to the correct one.
Conclusion and Future Work
This study presents a viable, low-cost solution for touchless text entry. While highly accurate, future iterations could be improved by adding "click" functionality (such as a long stare or eye gesture) to provide better control. Additionally, the authors suggest that 3D modeling of user viewpoints and multimodal integration with eye-tracking or BCI could further enhance the system's robustness and usability in diverse environments.
Summary updated