LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition

Adilson Medronha, Luís Lima, Janaína Claudio, Lucas S. Kupssinskü, Rodrigo C. Barros

Introduction

Sign Language Recognition (SLR) has seen significant advancements through deep learning, yet a "usability gap" persists. Most state-of-the-art models are computationally intensive, requiring high-end hardware or specialized sensors like depth cameras, which limits their accessibility for the general public. Furthermore, there is a lack of engaging tools to help the hearing community learn and practice dactylology (fingerspelling).

To address this, we introduce LERMO, a browser-based game inspired by the popular Wordle format. LERMO provides a low-friction, gamified environment for practicing Brazilian Sign Language (LIBRAS) using only a standard webcam. By optimizing for efficiency and real-time performance, LERMO makes sign language education accessible on standard consumer hardware.

LERMO gameplay interface showing the standard keyboard mode (c) and the LIBRAS camera-based mode (d).

System Architecture

The LERMO architecture is designed for high efficiency and low latency to ensure a seamless in-browser experience.

The Recognition Pipeline

The system utilizes a client-side Angular frontend. The core recognition process involves:

  1. Landmark Detection: MediaPipe is used to detect 21 hand landmarks from the webcam feed in real-time.
  2. Feature Engineering: To maintain a lightweight footprint, the system uses only the $x$ and $y$ coordinates (42 features), discarding the less reliable $z$ (depth) coordinate. Landmarks are scaled and centered relative to the wrist to ensure positional invariance.
  3. Classification: A Multi-Layer Perceptron (MLP) was selected after benchmarking against SVM, KNN, and Random Forest models. The final model is exceptionally compact (100KB, ~27,000 parameters), enabling fast delivery via onnxruntime-web.
The real-time hand detection pipeline using MediaPipe to extract landmarks from video frames.

Temporal Stability

To mitigate motion blur and landmark "jitter," LERMO employs a temporal filtering strategy. The system accumulates predictions over a 100-frame window and outputs the statistical mode as the final classification. At 20 FPS, this provides a stable result every 5 seconds, significantly improving user experience compared to raw, frame-by-frame predictions.

Data Collection and Training

The authors developed a custom landmark dataset for LIBRAS fingerspelling. Data was collected from five actors, yielding 500 samples per class across 25 letters and two control gestures ("delete" and "send").

To enhance model robustness, the dataset was augmented with horizontal flipping (supporting both left- and right-handed players), rotation, and resizing. Notably, the inclusion of an actor with limited finger joint mobility helped the model generalize to users with varying physical abilities.

Results and User Reception

The MLP model achieved a peak accuracy of 90.17% and an F1-score of 88.46%. Performance evaluations confirmed the model's efficiency; only 10.8% of surveyed users reported lag, validating the hardware-agnostic design.

In a user study of approximately 40 participants, 95% agreed that LERMO is a valuable tool for enhancing fingerspelling skills. Users praised the intuitive interface and the "hands-free" control signs (thumbs up/down), which allow gameplay without touching a keyboard.

Conclusion and Future Work

LERMO demonstrates that lightweight, well-optimized AI models can bridge the gap between research and practical educational tools. Future work will focus on optimizing the UI for mobile devices, expanding to word-level dynamic sign recognition, and adding support for other sign languages such as American Sign Language (ASL).

Summary updated

BibTeX
@inproceedings{medronha2024lermo,
  author = {Adilson Medronha and Luís Lima and Janaína Claudio and Lucas S. Kupssinskü and Rodrigo C. Barros},
  title = {{LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30384},
}
Figure