Smartphone video nystagmography using convolutional neural networks: ConVNG

Maximilian Friedrich, Erich Schneider, Miriam Buerklein, Johannes Taeger, Johannes Hartig, Jens Volkmann, Robert L. Peach, Daniel ZellerJournal of Neurology 2022 · DOI

TL;DR

This paper introduces ConVNG, a novel framework that uses convolutional neural networks (CNNs) to analyze smartphone videos for nystagmus detection. ConVNG achieves accuracy comparable to traditional, expensive videooculography (VOG) systems and significantly better precision than other computer vision benchmarks, paving the way for accessible, at-home eye movement diagnostics.

What Problem Does This Paper Address?

Eye movement abnormalities are common indicators of neurological disorders, but current diagnostic methods, like videooculography (VOG), are resource-intensive and not widely accessible. This limits their use in routine clinical practice, especially for conditions with episodic symptoms like vertigo, where timely diagnosis is crucial. The authors aim to bridge this gap by developing a low-cost, accessible solution for quantitative eye movement analysis using readily available smartphone technology. They specifically focus on nystagmus, a key symptom in many neurological conditions, and its primary diagnostic measure, slow-phase velocity (SPV).

Methods

The core of ConVNG is a fine-tuned convolutional neural network (CNN) based on the ResNet 50 architecture. This CNN is trained to track pupil outlines and key facial landmarks. The training involved over 550 annotated frames from diverse, real-world videos, capturing various lighting conditions, camera angles, and individuals, including those with pathological eye movements.

For analysis, videos are downsampled to 640x480 resolution, as this was found to be optimal for tracking performance. The system then derives kinematic measurements, including nystagmus direction and slow-phase velocity (SPV), using standard signal processing techniques like Fourier transforms and gradient calculations. Equivalence testing (TOST and Bayesian interval-null) was employed to compare ConVNG's performance against the gold-standard infrared VOG and other computer vision benchmarks like OpenFace and MediaPipe.

Datasets

  • Training Data: Over 550 annotated frames extracted from more than 50 highly variable videos depicting approximately 40 individuals. These videos were sourced from the authors' own collections and openly available resources, showcasing diverse real-world scenarios.
  • Independent Test Set: 10 videos, randomly sampled from the training data pool, used for evaluating the CNN's out-of-sample performance.
  • Clinical Validation Data:
    • Prospective: 10 healthy subjects (5 male, 5 female) aged 25-44 years.
  • Retrospective: Two clinical cases with existing VOG and smartphone video data: one with pendular nystagmus and one with downbeat nystagmus.
  • Benchmarks: OpenFace and MediaPipe were used as computer vision benchmarks for comparison.

Key Insights and Ideas

  • Democratizing Diagnostics: The most significant insight is that high-quality eye movement analysis, comparable to specialized clinical equipment, can be achieved using ubiquitous smartphone cameras. This has profound implications for remote patient monitoring and expanding access to neurological diagnostics.
  • CNNs for Clinical Nuance: Fine-tuning a CNN on clinically relevant, albeit diverse, video data is crucial. This approach allows the model to learn subtle patterns and handle real-world variations (like face masks, lighting changes) that generic computer vision models might struggle with.
  • SPV as a Robust Metric: The study reinforces the importance of SPV as a primary diagnostic metric for nystagmus and demonstrates that it can be reliably extracted from smartphone videos.
  • Precision Matters: While accuracy is important, the paper highlights that ConVNG's superior precision over benchmarks like MediaPipe is a key advantage for clinical utility, allowing for more sensitive detection of subtle changes.

Key Results

  • Accuracy: ConVNG tracking accuracy was estimated to be 9-15% of an average pupil diameter.
  • SPV Measurement: ConVNG achieved equivalence to VOG in SPV measurement accuracy (TOST p<0.017; Bayes factors > 24). It was the only method to achieve equivalence across all SPV calculations.
  • Precision:
    • ConVNG median precision: 0.30°/s.
  • MediaPipe median precision: 0.7°/s.
  • VOG median precision: 0.12°/s. ConVNG precision was significantly higher than MediaPipe, though VOG remained the most precise.

  • Model Robustness: The trained CNN demonstrated high out-of-sample robustness, with a median pupil marker likelihood of 0.85 in independent clinical videos.

  • Clinical Cases: Retrospective validation showed ConVNG's ability to derive kinematic descriptors with low relative deviations (9% and 5%) compared to VOG in real-world clinical scenarios.

Limitations and Future Work

  • Sequential Recording: ConVNG currently requires sequential recording sessions, as it cannot track pupils when obscured by VOG goggles.
  • Camera Stabilization: While not a limitation of the algorithm itself, camera stabilization is noted as important for optimal trace quality, especially in handheld recordings.
  • Low Light/Contrast: The approach, relying on RGB videos, is not expected to perform well in very low light or low contrast conditions.
  • Limited Velocity Range: The study focused on SPV values typical in clinical practice, not an exhaustive range of nystagmus velocities.
  • Future Work: The authors suggest large-scale prospective studies in real-world clinical settings to further validate ConVNG's robustness and reliability. They also emphasize the potential for this framework to be integrated into accessible smartphone applications for at-home monitoring and personalized medicine. The pretrained ConVNG model is made publicly available.

Summary updated

Abstract

ConVNG enables offline smartphone video nystagmography with an accuracy comparable to VOG and significantly higher precision than MediaPipe, a benchmark computer vision application for gaze estimation. This serves as a blueprint for highly accessible tools with potential to accelerate progress toward precise and personalized Medicine.
BibTeX
@article{friedrich2022smartphone,
  author = {Maximilian Friedrich and Erich Schneider and Miriam Buerklein and Johannes Taeger and Johannes Hartig and Jens Volkmann and Robert L. Peach and Daniel Zeller},
  title = {{Smartphone video nystagmography using convolutional neural networks: ConVNG}},
  journal = {Journal of Neurology},
  year = {2022},
  doi = {10.1007/s00415-022-11493-1},
}
Figure