Extracting decision-making features from the unstructured eye movements of clinicians on glaucoma OCT reports and developing AI models to classify expertise

Michelle Akerman, Sanmati Choudhary, Jeffrey M. Liebmann, George A. Cioffi, Royce W. S. Chen, Kaveri A. ThakoorFrontiers in Medicine 2023 · DOI

Abstract

This study investigates the eye movement patterns of ophthalmologists with varying expertise levels during the assessment of optical coherence tomography (OCT) reports for glaucoma detection. By recording gaze metrics from 13 participants (residents, fellows, and faculty), the researchers derived novel features using Sequence Graph Transform (SGT) to develop binary classification models. The results demonstrate that expert ophthalmologists exhibit more refined and focused fixations on critical diagnostic regions. AI models trained on these gaze patterns achieved up to 94.0% accuracy in differentiating between expert and novice clinicians, suggesting that eye-tracking can effectively "fingerprint" clinical expertise.

Introduction

Glaucoma is a leading cause of irreversible blindness, and early detection often hinges on interpreting complex Optical Coherence Tomography (OCT) reports. However, interpreting these reports is notoriously subjective; there is no universal "gold standard" for what features a clinician should prioritize. This study asks a different question: Can we model the human expert? By understanding the "search strategy" of a seasoned specialist, we can improve medical education for residents and build more "human-like" diagnostic AI. The challenge lies in turning unstructured, "messy" eye movements into structured data that a machine can understand.

Methods

The researchers recruited 13 clinicians from the Columbia University Irving Medical Center, ranging from junior residents to faculty experts. Participants diagnosed 20 digital Topcon swept-source OCT reports (10 healthy, 10 glaucomatous) while their gaze was recorded using Pupil Labs Core eye trackers.

Participant demographics and expertise grouping.

The technical pipeline involved:

  1. Region of Interest (ROI) Mapping: The OCT report was divided into seven functional zones (e.g., RNFL thickness maps, probability charts).
  2. Sequence Graph Transform (SGT): This is the study's novel feature-engineering step. Instead of just looking at where someone looked, SGT creates an embedding that captures the sequence and relationship between fixations, turning a temporal stream of gaze data into a high-dimensional feature vector.
Raw sequence of fixation IDs across different regions of interest used for SGT.
  1. Supervised Learning: They trained simple feed-forward neural networks to classify the viewer as "Expert" or "Novice" and to classify the report as "Healthy" or "Glaucomatous" based only on the expert's gaze.

Key Insights and Results

  • The Expertise Fingerprint: The model achieved 94.0% accuracy (AUC 0.94) in differentiating between expert and novice clinicians based on their eye movements. This suggests that "expertise" is a distinct, measurable physical behavior.
  • Validation of Clinical Heuristics: The data showed that experts prioritize the RNFL probability map and the circumpapillary RNFL b-scan. This statistically validates the "Columbia University (CU) method," a clinical heuristic that teaches residents to check the probability map first and use b-scans for confirmation.
  • Efficiency of Expertise: Experts are "lean" viewers, making significantly fewer fixations than novices but focusing more intensely on high-value areas.
  • Gaze as a Proxy for Difficulty: Experts took longer and made more fixations on glaucomatous reports than healthy ones. Novices showed no significant difference, suggesting they may not yet recognize which cases require deeper scrutiny.
  • Disease Detection via Gaze: When using only the gaze patterns of experts, the model could identify glaucomatous reports with 64.67% accuracy. While lower than image-based AI, it significantly outperforms chance, proving that expert gaze carries a signal of the underlying pathology.

Conclusion and Future Work

Unsupervised analysis revealed that senior residents’ gaze patterns are much more similar to experts than to junior residents, providing a potential metric for tracking resident progress. While limited by a small sample size, these findings have implications for enhancing ophthalmic education and developing effective diagnostic tools. Future research aims to incorporate fixation duration and saccade velocity into these models to create interactive training tools for ophthalmology residents.

Summary updated

Abstract

This study aimed to investigate the eye movement patterns of ophthalmologists with varying expertise levels during the assessment of optical coherence tomography (OCT) reports for glaucoma detection. Objectives included evaluating eye gaze metrics and patterns as a function of ophthalmic education, deriving novel features from eye-tracking, and developing binary classification models for disease detection and expertise differentiation. Thirteen ophthalmology residents, fellows, and clinicians specializing in glaucoma participated in the study. Junior residents had less than 1 year of experience, while senior residents had 2-3 years of experience. The expert group consisted of fellows and faculty with over 3 to 30+ years of experience. Each participant was presented with a set of 20 Topcon OCT reports (10 healthy and 10 glaucomatous) and was asked to determine the presence or absence of glaucoma and rate their confidence of diagnosis. The eye movements of each participant were recorded as they diagnosed the reports using a Pupil Labs Core eye tracker. Expert ophthalmologists exhibited more refined and focused eye fixations, particularly on specific regions of the OCT reports, such as the retinal nerve fiber layer (RNFL) probability map and circumpapillary RNFL b-scan. The binary classification models developed using the derived features demonstrated high accuracy up to 94.0% in differentiating between expert and novice clinicians. The derived features and trained binary classification models hold promise for improving the accuracy of glaucoma detection and distinguishing between expert and novice ophthalmologists. These findings have implications for enhancing ophthalmic education and for the development of effective diagnostic tools.
BibTeX
@inproceedings{akerman2023extracting,
  author = {Michelle Akerman and Sanmati Choudhary and Jeffrey M. Liebmann and George A. Cioffi and Royce W. S. Chen and Kaveri A. Thakoor},
  title = {{Extracting decision-making features from the unstructured eye movements of clinicians on glaucoma OCT reports and developing AI models to classify expertise}},
  booktitle = {Frontiers in Medicine},
  year = {2023},
  doi = {10.3389/fmed.2023.1251183},
}
Figure