Comparison of Machine Learning Approaches to Improve Diagnosis of Optic Neuropathy Using Photopic Negative Response Measured Using a Handheld Device

Tina Diao, Fareshta Kushzad, Megh Patel, Megha Bindiganavale, Munam Wasi, Mykel J. Kochenderfer, Heather E. MossFrontiers in Medicine 2021 · DOI

TL;DR

This paper explores using machine learning, specifically time series classification, to improve the diagnosis of optic neuropathies (ON) from electroretinogram (ERG) signals captured by a portable, handheld device. By analyzing the entire ERG waveform rather than just specific features, they achieved significantly better diagnostic performance, paving the way for more accessible and objective ON detection.

What Problem Does This Paper Address?

Optic neuropathies (ON) are a group of diseases that damage the optic nerve, leading to vision loss. Current diagnostic methods, like visual acuity tests and visual field perimetry, are often subjective and can fluctuate, making them less reliable for early detection and monitoring. The photopic negative response (PhNR), a component of the full-field electroretinogram (ERG), is known to be reduced in ON. However, traditional ERG measurements require specialized equipment and expertise, limiting its widespread clinical use. While newer handheld devices make ERG recording more accessible, classifying ON based on pre-defined ERG features (like PhNR amplitude) has shown only moderate success. This paper aims to bridge this gap by investigating whether advanced machine learning techniques, specifically time series classification, can leverage the full ERG waveform to achieve more accurate ON diagnosis.

Methods

The study employed two main approaches for analyzing ERG data:

  1. User-Defined ERG Features: Traditional methods involved extracting specific features from the ERG waveform, such as the photopic negative response (PhNR) amplitude at 72ms (PhNR72) and at its minimum (PhNRmin), as well as ratios like P-ratio and W-ratio. These features were then analyzed using generalized estimating equation (GEE) models to compare ON eyes with control eyes, accounting for age. Classification performance was assessed using receiver operating characteristic (ROC) curves.
  2. Time Series Classification (TSC): This is the core novel approach. Instead of relying on pre-selected features, TSC treats the entire ERG waveform as a sequence of data points over time. The researchers compared several established TSC algorithms, including:
  3. Nearest Neighbor Dynamic Time Warping (NN DTW)
    • Linear Support Vector Machine (SVM Linear)
    • Support Vector Machine with a Radial Basis Function Kernel (RBF Kernel SVM)
    • Random Forest (RF)
    • Gradient Boosting (GB)
    • Time Series Forest (TSF)
    • Long-Short Term Memory (LSTM) networks

These models were trained, validated, and tested on the ERG waveforms to predict whether an eye had ON or was a control.

Datasets

The study utilized ERG data collected from 217 eyes of 155 subjects. These subjects were recruited from a neuro-ophthalmology clinic and included:

  • Optic Neuropathy (ON+) eyes: 109 eyes diagnosed with various forms of ON (e.g., papilledema, compressive ON, optic neuritis).
  • Control (ON-) eyes: 108 eyes, comprising:
  • Control-fellow eyes (the fellow eye of a subject with unilateral ON).
  • Control-patient eyes (recruited from the clinic but without neuro-ophthalmic disease).
  • Healthy-control eyes (recruited from a vision research center with no known ophthalmic or neurological disease).

The data was split into training, validation, and testing sets to ensure robust model evaluation. The paper does not introduce a new benchmark dataset but rather uses data collected for this specific study.

Key Insights and Ideas

  • The Whole Waveform Matters: The most significant insight is that analyzing the entire ERG waveform using time series classification yields substantially better diagnostic performance than relying on pre-defined, user-extracted features. This suggests that the subtle temporal dynamics within the ERG signal hold crucial diagnostic information that is lost when only specific peaks or troughs are considered.
  • Handheld ERG is Viable for ML: The study demonstrates that even with a portable, handheld ERG device using skin electrodes (which can introduce noise and reduce signal amplitude), machine learning can extract meaningful diagnostic patterns. This validates the potential of these devices for broader clinical integration.
  • Ensemble Methods Shine: Among the TSC algorithms, ensemble methods like Time Series Forest (TSF) and Gradient Boosting (GB) showed particularly strong performance, outperforming more complex models like LSTMs in some metrics. This highlights the robustness and effectiveness of these approaches for this specific diagnostic task.
  • User-Defined Features are "Fair" at Best: While user-defined features showed statistically significant differences between ON and control eyes, their classification performance was only "fair," with AUCs generally below 0.7. This underscores the limitations of traditional feature engineering for this complex diagnostic problem.

Key Results

  • User-Defined Features: Classification based on user-defined ERG features resulted in a fair classification potential, with ROC AUCs ranging from 0.62 to 0.68 and F1 scores between 0.30 and 0.33.
  • Time Series Classification: Machine learning classifiers using time series analysis achieved significantly better results on a test dataset, with F1 scores ranging from 0.58 to 0.76.
  • The Time Series Forest (TSF) classifier achieved the highest precision (0.74), accuracy (0.74), and F1 score (0.76).
  • The RBF Kernel SVM achieved the highest recall (0.86).
  • PhNR Reduction in ON: Consistent with previous research, user-defined ERG measures like PhNR trough and at 72ms, P-ratio, and W-ratio were significantly lower in ON eyes compared to control eyes (p < 0.0005).

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the primary limitation is the relatively small dataset size. This restricted their ability to perform stratified analyses (e.g., by ON severity or covariates) and to split the data into independent training and testing sets for the user-defined feature analysis, potentially leading to an overestimation of performance for those methods. A larger sample size would also help refine the time series models.
  • ERG Protocol Limitations: The use of skin electrodes and recording in a dim room without light adaptation may have increased signal variability and reduced amplitude compared to traditional methods.
  • Future Directions: The authors suggest that larger sample sizes are crucial for further tuning and validating the time series models. They also propose that portable ERG devices combined with time series classification analysis hold significant promise for developing a bedside diagnostic test for ON. Further research could explore more advanced deep learning architectures or investigate the impact of different ERG recording protocols on classification performance.

Summary updated

Abstract

The photopic negative response of the full-field electroretinogram (ERG) is reduced in optic neuropathies. However, technical requirements for measurement and poor classification performance have limited widespread clinical application. Recent advances in hardware facilitate efficient clinic-based recording of the full-field ERG. Time series classification, a machine learning approach, may improve classification by using the entire ERG waveform as the input. In this study, full-field ERGs were recorded in 217 eyes (109 optic neuropathy and 108 controls) of 155 subjects. User-defined ERG features including photopic negative response were reduced in optic neuropathy eyes (<i>p</i> < 0.0005, generalized estimating equation models accounting for age). However, classification of optic neuropathy based on user-defined features was only fair with receiver operating characteristic area under the curve ranging between 0.62 and 0.68 and F1 score at the optimal cutoff ranging between 0.30 and 0.33. In comparison, machine learning classifiers using a variety of time series analysis approaches had F1 scores of 0.58-0.76 on a test data set. Time series classifications are promising for improving optic neuropathy diagnosis using ERG waveforms. Larger sample sizes will be important to refine the models.
BibTeX
@inproceedings{diao2021comparison,
  author = {Tina Diao and Fareshta Kushzad and Megh Patel and Megha Bindiganavale and Munam Wasi and Mykel J. Kochenderfer and Heather E. Moss},
  title = {{Comparison of Machine Learning Approaches to Improve Diagnosis of Optic Neuropathy Using Photopic Negative Response Measured Using a Handheld Device}},
  booktitle = {Frontiers in Medicine},
  year = {2021},
  doi = {10.3389/fmed.2021.771713},
}
Figure