Collaboration between clinicians and vision–language models in radiology report generation

Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles T. Lau, Tao Tu, Shekoofeh Azizi, K. K. Singhal, Mike Schaekermann, R. May, Roy Lee, SiWai Man, S. Sara Mahdavi, Zahra Ahmed, Yossi Matias, Joëlle Barral, S. M. Ali Eslami, Danielle Belgrave, Yun Liu, Sreenivasa Raju Kalidindi, Shravya Shetty, Vivek Natarajan, Pushmeet Kohli, Po-Sen Huang, Alan Karthikesalingam, Sofia Ira KtenaNature Medicine 2024 · DOI

Abstract

Automated radiology report generation has the potential to improve patient care and reduce the workload of radiologists. However, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of artificial intelligence (AI)-generated reports. We build a state-of-the-art report generation system for chest radiographs, called Flamingo-CXR, and perform an expert evaluation of AI-generated reports by engaging a panel of board-certified radiologists. We observe a wide distribution of preferences across the panel and across clinical settings, with 56.1% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for in/outpatient X-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. Errors were observed in human-written reports and Flamingo-CXR reports, with 24.8% of in/outpatient cases containing clinically significant errors in both report types, 22.8% in Flamingo-CXR reports only and 14.0% in human reports only. For reports that contain errors we develop an assistive setting, a demonstration of clinician-AI collaboration for radiology report composition, indicating new possibilities for potential clinical utility.

Introduction

The field of radiology is currently grappling with a global shortage of specialists, leading to significant delays in patient care and increased clinician burnout. While artificial intelligence (AI) has demonstrated proficiency in specific diagnostic tasks, the generation of comprehensive, nuanced radiology reports remains a significant challenge. The primary obstacle to integrating AI into clinical workflows is not merely text generation, but the rigorous evaluation of clinical quality. Traditional metrics like BLEU or CIDEr, which measure linguistic similarity to reference texts, often fail to account for medical accuracy. To address this, researchers from Google DeepMind developed Flamingo-CXR, a vision-language model designed to generate clinical-grade reports for chest radiographs, and established a human-centric evaluation framework involving board-certified radiologists.

Methods and Evaluation Framework

Flamingo-CXR is built upon the Flamingo architecture, utilizing a 400-million parameter model that integrates visual data from X-ray images with textual information. The architecture includes a vision encoder and a perceiver resampler, enabling the language model to process visual features while predicting report text. The model was fine-tuned using importance weighting to prioritize the detection of rare but critical abnormalities.

The study's evaluation framework moved beyond automated metrics to focus on two primary human-centric tasks:

  1. Pairwise Preference: Radiologists performed blinded comparisons between AI-generated and human-written reports to determine which was superior for patient care.
  2. Error Correction and Assistive Setting: Experts edited reports to identify "clinically significant" errors, demonstrating the potential for clinician-AI collaboration.
Evaluation framework for AI-generated radiology reports, including pairwise preference and error correction tasks.

The model was trained and validated on two diverse datasets: the public MIMIC-CXR dataset (emergency department cases from the US) and the private IND1 dataset (inpatient and outpatient cases from India). This geographic diversity allowed researchers to account for variations in reporting styles and disease prevalence.

Key Results and Clinical Accuracy

The evaluation revealed that AI-generated reports are often comparable to those written by clinicians. In the in/outpatient (IND1) dataset, 77.7% of Flamingo-CXR reports were rated as equivalent or preferable to human reports by at least half of the expert panel. This figure rose to 94% for cases with no abnormal findings. In the more complex intensive care (MIMIC) setting, 56.1% of reports met this threshold.

Quantitatively, Flamingo-CXR achieved a CheXpert $F_1$ score of 0.519, representing a 33% improvement over previous state-of-the-art models. The study also highlighted human fallibility: 14.0% of human-written reports in the in/outpatient dataset contained clinically significant errors that the AI did not make. Conversely, 22.8% of cases had errors unique to the AI report, suggesting that AI and humans possess complementary strengths and weaknesses.

Overall detection accuracy of Flamingo-CXR compared to individual radiologists.

The Potential of Clinician-AI Collaboration

A significant finding of the research is the utility of AI as a "first draft" tool. In the assistive setting, reports generated through clinician-AI collaboration—where radiologists edited AI-generated drafts—were significantly preferred over original human-only reports. This suggests that the most effective path toward clinical adoption is not full automation, but a collaborative framework that leverages AI to reduce workload while maintaining human oversight.

Limitations and Future Directions

Flamingo-CXR remains an experimental research model with several limitations. Currently, it processes only a single frontal X-ray, whereas clinical practice typically involves lateral views, prior scans, and comprehensive patient histories. The model also faces challenges with precise counting and complex spatial reasoning. Future research will focus on developing interactive AI systems that allow for bidirectional dialogue between radiologists and the model, further enhancing the collaborative diagnostic process.

Summary updated

Abstract

Automated radiology report generation has the potential to improve patient care and reduce the workload of radiologists. However, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of artificial intelligence (AI)-generated reports. We build a state-of-the-art report generation system for chest radiographs, called Flamingo-CXR, and perform an expert evaluation of AI-generated reports by engaging a panel of board-certified radiologists. We observe a wide distribution of preferences across the panel and across clinical settings, with 56.1% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for in/outpatient X-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. Errors were observed in human-written reports and Flamingo-CXR reports, with 24.8% of in/outpatient cases containing clinically significant errors in both report types, 22.8% in Flamingo-CXR reports only and 14.0% in human reports only. For reports that contain errors we develop an assistive setting, a demonstration of clinician-AI collaboration for radiology report composition, indicating new possibilities for potential clinical utility.
BibTeX
@inproceedings{tanno2024collaboration,
  author = {Ryutaro Tanno and David G. T. Barrett and Andrew Sellergren and Sumedh Ghaisas and Sumanth Dathathri and Abigail See and Johannes Welbl and Charles T. Lau and Tao Tu and Shekoofeh Azizi and K. K. Singhal and Mike Schaekermann and R. May and Roy Lee and SiWai Man and S. Sara Mahdavi and Zahra Ahmed and Yossi Matias and Joëlle Barral and S. M. Ali Eslami and Danielle Belgrave and Yun Liu and Sreenivasa Raju Kalidindi and Shravya Shetty and Vivek Natarajan and Pushmeet Kohli and Po-Sen Huang and Alan Karthikesalingam and Sofia Ira Ktena},
  title = {{Collaboration between clinicians and vision–language models in radiology report generation}},
  booktitle = {Nature Medicine},
  year = {2024},
  doi = {10.1038/s41591-024-03302-1},
}
Figure