Abstract
Deep learning has shown recent success in classifying anomalies in chest x-rays, but datasets remain small compared to natural image datasets. Supervision of abnormality localization has been shown to improve trained models, partially compensating for dataset sizes. However, explicitly labeling these anomalies requires an expert and is very time-consuming. We propose a potentially scalable method for collecting implicit localization data using an eye tracker to capture gaze locations and a microphone to capture a dictation of a report, imitating the setup of a reading room. The resulting REFLACX (Reports and Eye-Tracking Data for Localization of Abnormalities in Chest X-rays) dataset was labeled across five radiologists and contains 3,032 synchronized sets of eye-tracking data and timestamped report transcriptions for 2,616 chest x-rays from the MIMIC-CXR dataset. We also provide auxiliary annotations, including bounding boxes around lungs and heart and validation labels consisting of ellipses localizing abnormalities and image-level labels. Furthermore, a small subset of the data contains readings from all radiologists, allowing for the calculation of inter-rater scores.
Introduction: The Localization Bottleneck
Deep learning has revolutionized medical image analysis, yet its performance is often constrained by the availability of high-quality, spatially labeled data. While large-scale datasets exist for image-level classification (e.g., determining if a chest x-ray contains an abnormality), datasets that specify where an abnormality is located are significantly rarer.
Traditionally, "localization" labels require radiologists to manually draw bounding boxes or ellipses. This process is labor-intensive, expensive, and difficult to scale to the hundreds of thousands of images required for robust model training. The REFLACX dataset addresses this bottleneck by introducing a method for "implicit" localization: capturing the natural gaze and speech of a radiologist during their standard clinical workflow.
Methodology: The Implicit Labeling Workflow
To collect this data, researchers established a simulated "reading room" environment. Five board-certified radiologists participated in the study using a setup equipped with an Eyelink 1000 Plus eye tracker and a high-quality microphone.
The workflow was designed to minimize interference with the radiologists' natural diagnostic process:
- Dictation: The radiologist viewed a chest x-ray (CXR) and dictated a clinical report.
- Eye-Tracking: The system recorded gaze locations at 1,000 Hz, identifying "fixations"—the moments when the eye remains still to process visual information.
- Synchronization: Audio recordings were transcribed, and each word was timestamped. This allowed researchers to align specific spoken findings (e.g., "pleural effusion") with the spatial coordinates the radiologist was viewing at that moment.
- Validation: To provide a ground truth for comparison, radiologists also performed traditional explicit labeling, drawing ellipses around abnormalities and bounding boxes around the heart and lungs.
Dataset Characteristics
The REFLACX dataset is built upon the MIMIC-CXR collection, a large public repository of chest radiographs. The resulting dataset includes:
- 3,032 synchronized readings covering 2,616 unique chest x-rays.
- Ancillary Data: Manual ground truth ellipses, bounding boxes for anatomical normalization, and image-level labels with certainty scores.
- Inter-rater Reliability: A subset of images was reviewed by all five radiologists, providing Fleiss’ Kappa scores to quantify agreement across different types of abnormalities.
Key Insights and Results
The "Look-Ahead" Effect
A critical finding of the study is the temporal relationship between visual processing and verbalization. The data revealed a peak in correlation between gaze location and the mentioned abnormality approximately 2.5 seconds (roughly one sentence unit) before the radiologist actually spoke the finding. This "look-ahead" effect is vital for researchers attempting to align gaze data with report text.
Validation of Implicit Labels
The study demonstrated that a radiologist's eyes naturally "draw" localization labels. The Normalized Cross-Correlation (NCC) between eye fixations and manual ellipses was 0.380, significantly higher than the baseline of 0.326 (which represented typical areas radiologists look at regardless of findings). This confirms that gaze data contains specific, usable information for abnormality localization.
Scalability and Precision
The eye-tracking system maintained high spatial accuracy, with an average calibration error of less than 0.5 degrees of visual angle. Because the data collection mimics a standard clinical environment, this method could theoretically be deployed at scale in hospitals, allowing for the passive collection of millions of labeled images without requiring additional time from medical experts.
Limitations and Future Directions
While REFLACX represents a significant step forward, the authors noted several areas for future development:
- Clinical Environment: The study used a single-screen setup, whereas many clinical stations use multiple monitors, which may influence gaze patterns.
- View Constraints: The current dataset is limited to frontal (PA/AP) views; lateral views were not included in this phase.
- Alignment Complexity: The 2.5-second delay between looking and speaking necessitates the development of sophisticated temporal alignment algorithms to accurately map text to coordinates.
- Automation: Currently, the need for frequent recalibration and data quality checks requires a human coordinator. Automating these processes will be essential for true clinical integration.
Conclusion
REFLACX provides a unique, multi-modal resource for the medical AI community. By demonstrating that eye-tracking data can serve as a proxy for manual localization, the researchers have opened a path toward massive, passively labeled datasets that could significantly improve the accuracy and interpretability of deep learning models in radiology.
Summary updated