JeFaPaTo - A joint toolbox for blinking analysis and facial features extraction

Tim Büchner, Oliver Mothes, Orlando Guntinas-Lichius, Joachim DenzlerarXiv 2024 · DOI · arXiv

Abstract

JeFaPaTo (Jena Facial Palsy Toolbox) is an open-source Python toolbox designed to bring advanced computer vision to the clinical bedside. It provides a user-friendly interface for high-resolution blinking analysis and facial feature extraction, moving beyond simple "open vs. closed" eye detection to capture nuanced kinematics—like duration, velocity, and partial blinks—essential for diagnosing and monitoring conditions like facial palsy and Parkinson’s disease. By bridging the gap between advanced computer vision and clinical usability through a graphical user interface, it enables medical professionals to perform detailed temporal analysis on high-speed video (up to 240 FPS).

Introduction

Analyzing facial expressions is a cornerstone of neurology and otorhinolaryngology, yet it remains a manual or overly simplified task in many clinics. While computer vision has made leaps in facial landmark detection, a significant "usability gap" exists. Most state-of-the-art tools require programming expertise, making them inaccessible to medical professionals.

Furthermore, existing automated tools often treat blinking as a binary state: the eye is either open or closed. For a patient with facial nerve damage (facial palsy), the "in-between" matters most. Clinicians need to measure the synchronicity between the left and right eyes, the speed of the eyelid movement, and the occurrence of "partial blinks" where the eye doesn't close completely. JeFaPaTo addresses these needs by providing a standalone, GUI-based tool that handles high-temporal resolution video (up to 240 FPS) to capture movements that occur in less than 100 milliseconds.

Methods

JeFaPaTo is built on a Python-based architecture that offers both a programming interface for researchers and a graphical user interface (GUI) for clinicians. The toolbox leverages the MediaPipe library to extract 468 facial landmarks and 52 blend shape features in real-time.

To quantify eye closure, it calculates the Eye-Aspect-Ratio (EAR), which compares the vertical and horizontal distances between specific eye landmarks. The tool offers both 2D and 3D versions of EAR to help mitigate the effects of head rotation. Blink detection is achieved using the scipy.signal.find_peaks algorithm combined with Otsu’s method for automatic thresholding. This allows the software to distinguish between complete blinks, partial blinks, and "blink intensity."

The JeFaPaTo Interface

Built with PyQt6 and pyqtgraph, the interface provides a comprehensive environment for video analysis. The GUI allows experts to manually correct or validate automatically detected blinks, ensuring that the final medical statistics are reliable. This "human-in-the-loop" approach is vital for clinical applications where automated errors could lead to misdiagnosis.

The JeFaPaTo graphical user interface showing the EAR time series (A), blink table (B), algorithm settings (C), and video preview (D).

Key Insights and Clinical Utility

  • Temporal Resolution: Standard 30 FPS video is often insufficient for medical blink analysis. JeFaPaTo is optimized for 240 FPS, which is necessary to accurately characterize the acceleration and deceleration phases of a blink.
  • Accessibility: By providing standalone executables for Windows, Linux, and MacOS, the authors remove the technical barriers that often prevent clinical adoption of open-source research tools.
  • The "Partial Blink" Problem: By using prominence-based peak detection rather than a fixed threshold, the tool can identify blinks where the eyelid only moves halfway—a critical metric for assessing the severity of facial nerve impairment.

Results and Visual Summaries

The tool extracts a wide array of medically relevant statistics, including inter-blink delay, blink amplitude, and frequency per minute (BPM). It achieves a processing speed of 60 FPS for landmark extraction on standard modern hardware. A standout feature is the generation of "visual summaries" that capture long-term blinking behavior in a single view, allowing clinicians to quickly spot patterns or anomalies in synchronicity between the left and right eyes.

A visual summary generated by JeFaPaTo, illustrating blink frequency (top), temporal distribution of EAR deviations (middle), and a histogram of deviations (right) over a 20-minute recording.

Limitations and Future Work

While the 3D-EAR feature helps, the EAR metric remains sensitive to significant head movements. The authors plan to implement 3D head pose estimation to further correct EAR scores in future versions. Additionally, they aim to integrate joint analysis of eye and mouth movements to better quantify "synkinesis"—involuntary muscle movements common in facial palsy.

Summary updated

Abstract

Analyzing facial features and expressions is a complex task in computer vision. The human face is intricate, with significant shape, texture, and appearance variations. In medical contexts, facial structures and movements that differ from the norm are particularly important to study and require precise analysis to understand the underlying conditions. Given that solely the facial muscles, innervated by the facial nerve, are responsible for facial expressions, facial palsy can lead to severe impairments in facial movements. One affected area of interest is the subtle movements involved in blinking. It is an intricate spontaneous process that is not yet fully understood and needs high-resolution, time-specific analysis for detailed understanding. However, a significant challenge is that many computer vision techniques demand programming skills for automated extraction and analysis, making them less accessible to medical professionals who may not have these skills. The Jena Facial Palsy Toolbox (JeFaPaTo) has been developed to bridge this gap. It utilizes cutting-edge computer vision algorithms and offers a user-friendly interface for those without programming expertise. This toolbox makes advanced facial analysis more accessible to medical experts, simplifying integration into their workflow.
BibTeX
@misc{buchner2024jefapato,
  author = {Tim Büchner and Oliver Mothes and Orlando Guntinas-Lichius and Joachim Denzler},
  title = {{JeFaPaTo - A joint toolbox for blinking analysis and facial features extraction}},
  year = {2024},
  eprint = {2402.08439},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.08439},
}
Figure