American Sign Language Recognition Using an FMCW Wireless Sensor (Student Abstract)

Yuanqi Du, Nguyen Dang, Riley Wilkerson, Parth H. Pathak, Huzefa Rangwala, Jana Kosecka

Introduction

Translating American Sign Language (ASL) into text or speech is a critical step toward breaking down communication barriers. Historically, researchers have relied on computer vision (cameras) or wearable sensors (smart gloves). However, cameras raise significant privacy concerns and are sensitive to lighting, while wearables are physically invasive and uncomfortable for daily use.

Researchers at George Mason University have developed a "middle ground" using Frequency-Modulated Continuous-Wave (FMCW) radar. This non-invasive, privacy-preserving technology tracks 3D point clouds to recognize gestures without capturing identifiable images or requiring the user to wear specialized equipment.

Methodology

The system utilizes an FMCW wireless sensor to emit signals that bounce off the user, generating thousands of 3D "cloud points" per second. To transform this raw data into recognizable signs, the team implemented a specialized processing pipeline.

The "Cell Division" Algorithm

Rather than tracking individual points—which can be noisy and flicker—the researchers partitioned a virtual 3D cube around the user into equal "cells" (voxels). By measuring the occupancy of these cells (the percentage of total points within each cell), the system creates a stable feature set that is resilient to signal noise.

Figure 2: 3D point cloud representation of a user, identifying the body and hands.

Temporal Tracking and Classification

To capture the motion of a gesture, a sliding window technique was employed to combine multiple time frames, allowing the model to observe the full sequence of a sign from beginning to end. Noise was filtered using DBSCAN (a density-based clustering algorithm), and a variance threshold was applied to ignore static cells. For classification, the team tested several models, finding that a Multilayer Perceptron (MLP) yielded the best results.

Key Insights: Privacy and Occupancy

A primary advantage of this radar-based approach is Privacy by Design. Because the sensor only captures point clouds, it is impossible to identify the user's face or surroundings, making it a socially acceptable technology for home use.

Figure 6: Comparison between visual camera frames and radar point cloud sequences for a gesture.

Furthermore, the shift from tracking specific points to measuring cell occupancy makes the system much more robust. However, the researchers noted that sign language is highly individual; these "idiosyncrasies" mean that models often require user-specific tuning to maintain high accuracy across different people.

Results and Future Work

The system achieved an impressive 95% accuracy for single-user gesture prediction across a vocabulary of 19 common ASL signs. While cross-user accuracy is more challenging, the team found that incorporating just 30% of a new user's data into the training set could boost performance to 80% accuracy.

Future work will focus on improving hand localization to better distinguish between similar gestures and developing more complex models to capture fine-grained finger movements.

Summary updated

Abstract

In today's digital world, rapid technological advancements continue to lessen the burden of tasks for individuals. Among these tasks is communication across perceived language barriers. Indeed, increased attention has been drawn to American Sign Language (ASL) recognition in recent years. Camera-based and motion detection-based methods have been researched extensively; however, there remains a divide in communication between ASL users and non-users. Therefore, this research team proposes the use of a novel wireless sensor (Frequency-Modulated Continuous-Wave Radar) to help bridge the gap in communication. In short, this device sends out signals that detect the user's body positioning in space. These signals then reflect off the body and back to the sensor, developing thousands of cloud points per second, indicating where the body is positioned in space. These cloud points can then be examined for movement over multiple consecutive time frames using a cell division algorithm, ultimately showing how the body moves through space as it completes a single gesture or sentence. At the end of the project, 95% accuracy was achieved in one-object prediction as well as 80% accuracy on cross-object prediction with 30% other objects' data introduced on 19 commonly used gestures. There are 30 samples for each gesture per person from three persons.
BibTeX
@inproceedings{du2020american,
  author = {Yuanqi Du and Nguyen Dang and Riley Wilkerson and Parth H. Pathak and Huzefa Rangwala and Jana Kosecka},
  title = {{American Sign Language Recognition Using an FMCW Wireless Sensor (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7162},
}
Figure