TL;DR
This paper explores using machine learning, specifically time series classification, to improve the diagnosis of optic neuropathies (ON) from electroretinogram (ERG) signals captured by a portable, handheld device. By analyzing the entire ERG waveform rather than just specific features, they achieved significantly better diagnostic performance, paving the way for more accessible and objective ON detection.
What Problem Does This Paper Address?
Optic neuropathies (ON) are a group of diseases that damage the optic nerve, leading to vision loss. Current diagnostic methods, like visual acuity tests and visual field perimetry, are often subjective and can fluctuate, making them less reliable for early detection and monitoring. The photopic negative response (PhNR), a component of the full-field electroretinogram (ERG), is known to be reduced in ON. However, traditional ERG measurements require specialized equipment and expertise, limiting its widespread clinical use. While newer handheld devices make ERG recording more accessible, classifying ON based on pre-defined ERG features (like PhNR amplitude) has shown only moderate success. This paper aims to bridge this gap by investigating whether advanced machine learning techniques, specifically time series classification, can leverage the full ERG waveform to achieve more accurate ON diagnosis.
Methods
The study employed two main approaches for analyzing ERG data:
- User-Defined ERG Features: Traditional methods involved extracting specific features from the ERG waveform, such as the photopic negative response (PhNR) amplitude at 72ms (PhNR72) and at its minimum (PhNRmin), as well as ratios like P-ratio and W-ratio. These features were then analyzed using generalized estimating equation (GEE) models to compare ON eyes with control eyes, accounting for age. Classification performance was assessed using receiver operating characteristic (ROC) curves.
- Time Series Classification (TSC): This is the core novel approach. Instead of relying on pre-selected features, TSC treats the entire ERG waveform as a sequence of data points over time. The researchers compared several established TSC algorithms, including:
- Nearest Neighbor Dynamic Time Warping (NN DTW)
- Linear Support Vector Machine (SVM Linear)
- Support Vector Machine with a Radial Basis Function Kernel (RBF Kernel SVM)
- Random Forest (RF)
- Gradient Boosting (GB)
- Time Series Forest (TSF)
- Long-Short Term Memory (LSTM) networks
These models were trained, validated, and tested on the ERG waveforms to predict whether an eye had ON or was a control.
Datasets
The study utilized ERG data collected from 217 eyes of 155 subjects. These subjects were recruited from a neuro-ophthalmology clinic and included:
- Optic Neuropathy (ON+) eyes: 109 eyes diagnosed with various forms of ON (e.g., papilledema, compressive ON, optic neuritis).
- Control (ON-) eyes: 108 eyes, comprising:
- Control-fellow eyes (the fellow eye of a subject with unilateral ON).
- Control-patient eyes (recruited from the clinic but without neuro-ophthalmic disease).
- Healthy-control eyes (recruited from a vision research center with no known ophthalmic or neurological disease).
The data was split into training, validation, and testing sets to ensure robust model evaluation. The paper does not introduce a new benchmark dataset but rather uses data collected for this specific study.
Key Insights and Ideas
- The Whole Waveform Matters: The most significant insight is that analyzing the entire ERG waveform using time series classification yields substantially better diagnostic performance than relying on pre-defined, user-extracted features. This suggests that the subtle temporal dynamics within the ERG signal hold crucial diagnostic information that is lost when only specific peaks or troughs are considered.
- Handheld ERG is Viable for ML: The study demonstrates that even with a portable, handheld ERG device using skin electrodes (which can introduce noise and reduce signal amplitude), machine learning can extract meaningful diagnostic patterns. This validates the potential of these devices for broader clinical integration.
- Ensemble Methods Shine: Among the TSC algorithms, ensemble methods like Time Series Forest (TSF) and Gradient Boosting (GB) showed particularly strong performance, outperforming more complex models like LSTMs in some metrics. This highlights the robustness and effectiveness of these approaches for this specific diagnostic task.
- User-Defined Features are "Fair" at Best: While user-defined features showed statistically significant differences between ON and control eyes, their classification performance was only "fair," with AUCs generally below 0.7. This underscores the limitations of traditional feature engineering for this complex diagnostic problem.
Key Results
- User-Defined Features: Classification based on user-defined ERG features resulted in a fair classification potential, with ROC AUCs ranging from 0.62 to 0.68 and F1 scores between 0.30 and 0.33.
- Time Series Classification: Machine learning classifiers using time series analysis achieved significantly better results on a test dataset, with F1 scores ranging from 0.58 to 0.76.
- The Time Series Forest (TSF) classifier achieved the highest precision (0.74), accuracy (0.74), and F1 score (0.76).
- The RBF Kernel SVM achieved the highest recall (0.86).
- PhNR Reduction in ON: Consistent with previous research, user-defined ERG measures like PhNR trough and at 72ms, P-ratio, and W-ratio were significantly lower in ON eyes compared to control eyes (p < 0.0005).
Limitations and Future Work
- Dataset Size: The authors acknowledge that the primary limitation is the relatively small dataset size. This restricted their ability to perform stratified analyses (e.g., by ON severity or covariates) and to split the data into independent training and testing sets for the user-defined feature analysis, potentially leading to an overestimation of performance for those methods. A larger sample size would also help refine the time series models.
- ERG Protocol Limitations: The use of skin electrodes and recording in a dim room without light adaptation may have increased signal variability and reduced amplitude compared to traditional methods.
- Future Directions: The authors suggest that larger sample sizes are crucial for further tuning and validating the time series models. They also propose that portable ERG devices combined with time series classification analysis hold significant promise for developing a bedside diagnostic test for ON. Further research could explore more advanced deep learning architectures or investigate the impact of different ERG recording protocols on classification performance.
Summary updated