REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays

Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F. Auffermann, Jessica Chan, Phuong-Anh T. Duong, Vivek Srikumar, Trafton Drew, Joyce Schroeder, Tolga Taşdizen

TL;DR

REFLACX is a novel dataset that pairs over 2,600 chest x-rays with synchronized eye-tracking data and timestamped radiology reports. By capturing where a radiologist looks while they dictate, the researchers demonstrate a scalable, "implicit" method for generating the localization labels (where exactly the abnormality is) that deep learning models desperately need.

What Problem Does This Paper Address?

Deep learning has revolutionized medical image analysis, but it has a massive appetite for labeled data. While we have large datasets for classifying whether a chest x-ray (CXR) has an abnormality, we have very few that show where it is.

Currently, "localization" labels require experts to manually draw bounding boxes or ellipses—a process that is incredibly time-consuming and expensive. This creates a bottleneck: we have hundreds of thousands of images, but only a tiny fraction have the high-quality spatial labels needed to train models that can accurately "point" to a disease. The authors of REFLACX propose a way to bypass this by capturing labels implicitly during a radiologist’s natural workflow.

Methods

The researchers set up a "reading room" environment equipped with an Eyelink 1000 Plus eye tracker and a microphone. Five board-certified radiologists participated in the study.

The workflow was designed to be non-intrusive:

  1. Dictation: The radiologist views a CXR and dictates a report as they would in clinical practice.
  2. Eye-Tracking: The system records gaze locations at 1,000 Hz, capturing "fixations" (where the eye stays still to process information).
  3. Synchronization: The audio is transcribed, and each word is timestamped. This allows the researchers to link specific spoken observations (e.g., "pleural effusion") to the exact coordinates the doctor was looking at moments before.
  4. Validation Labels: To ensure the eye-tracking data was actually useful, radiologists also performed a traditional "explicit" labeling task, drawing ellipses around abnormalities and bounding boxes around the heart and lungs.

Datasets

  • Source Material: The images were sampled from the MIMIC-CXR dataset, a massive, publicly available collection of chest radiographs.
  • REFLACX Dataset: Contains 3,032 synchronized sets of eye-tracking data and transcriptions for 2,616 unique chest x-rays.
  • Ancillary Data: Includes manual "ground truth" ellipses for abnormalities, bounding boxes for lung/heart normalization, and image-level labels with certainty scores.

Key Insights and Ideas

  • The "Look-Ahead" Effect: One of the most fascinating findings is the temporal relationship between sight and speech. The data shows a peak in correlation between gaze and abnormality location about 2.5 seconds (or roughly one sentence unit) before the radiologist actually mentions the abnormality in their report.
  • Implicit vs. Explicit: The paper proves that a doctor's eyes "draw" the labels naturally. The Normalized Cross-Correlation (NCC) between eye fixations and manual ellipses was significantly higher than a baseline of "usual" areas radiologists look at, confirming that gaze carries specific, usable localization information.
  • Scalability as a Feature: Because this method mimics a standard clinical setup, it could theoretically be deployed in real radiology departments. This would allow for the "passive" collection of millions of labeled images without requiring doctors to change their behavior or spend extra time on annotations.

Key Results

  • Dataset Scale: 3,032 readings across three phases, providing a rich resource for multi-modal machine learning.
  • High Precision: The eye-tracking system maintained an average calibration error of less than 0.5 degrees of visual angle, ensuring the gaze data is spatially accurate.
  • Validation: The researchers achieved an average NCC of 0.380 for fixation heatmaps against labeled ellipses, outperforming the baseline (0.326).
  • Inter-rater Reliability: The paper provides Fleiss’ Kappa scores for various abnormalities, offering a benchmark for how much radiologists agree on specific findings (e.g., high agreement on "Enlarged cardiac silhouette," lower on "Airway wall thickening").

Limitations and Future Work

  • Simplified Setup: The study used a single-screen setup, whereas clinical environments often use multiple monitors. This might change gaze behavior.
  • Frontal Only: The dataset is limited to frontal (PA/AP) views; lateral views were not included.
  • Temporal Alignment: Because of the ~2.5-second delay between looking and speaking, future researchers will need to develop sophisticated alignment algorithms to perfectly map words to gaze coordinates.
  • Clinical Integration: While promising, the authors note that the "cost" of frequent recalibration (every 45-60 minutes) and the need for high-quality data checking currently require a human coordinator, which needs to be automated for true clinical scale.

Summary updated

Abstract

Deep learning has shown recent success in classifying anomalies in chest x-rays, but datasets are still small compared to natural image datasets. Supervision of abnormality localization has been shown to improve trained models, partially compensating for dataset sizes. However, explicitly labeling these anomalies requires an expert and is very time-consuming. We propose a potentially scalable method for collecting implicit localization data using an eye tracker to capture gaze locations and a microphone to capture a dictation of a report, imitating the setup of a reading room. The resulting REFLACX (Reports and Eye-Tracking Data for Localization of Abnormalities in Chest X-rays) dataset was labeled across five radiologists and contains 3,032 synchronized sets of eye-tracking data and timestamped report transcriptions for 2,616 chest x-rays from the MIMIC-CXR dataset. We also provide auxiliary annotations, including bounding boxes around lungs and heart and validation labels consisting of ellipses localizing abnormalities and image-level labels. Furthermore, a small subset of the data contains readings from all radiologists, allowing for the calculation of inter-rater scores.
BibTeX
@inproceedings{lanfredi2022reflacx,
  author = {Ricardo Bigolin Lanfredi and Mingyuan Zhang and William F. Auffermann and Jessica Chan and Phuong-Anh T. Duong and Vivek Srikumar and Trafton Drew and Joyce Schroeder and Tolga Taşdizen},
  title = {{REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41597-022-01441-z},
}

Loading related content...

Figure