BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports

Yuxuan Chen, Haoyan Yang, Hengkai Pan, Fardeen Siddiqui, Antonio Verdone, Qingyang Zhang, Sumit Chopra, Chen Zhao, Yiqiu Shen

Introduction

Radiology reports for breast ultrasounds are cornerstone documents in cancer screening and diagnosis. While radiologists adhere to standardized guidelines like the Breast Imaging-Reporting and Data System (BI-RADS), the resulting reports are typically documented in free-text prose. This unstructured format presents a significant barrier to automated clinical decision support, large-scale research, and the tracking of patient outcomes.

While large-scale proprietary models such as GPT-4 have demonstrated high proficiency in extracting structured data from medical text, they pose two major challenges for healthcare institutions: data privacy and operational cost. Sending Protected Health Information (PHI) to external cloud-based APIs raises compliance risks, and the cost of processing thousands of reports can be prohibitive. To address these issues, researchers developed BURExtract-Llama, a specialized, open-source model designed to match the performance of state-of-the-art LLMs while operating securely within a hospital's local infrastructure.

Methodology

The development of BURExtract-Llama followed a "teacher-student" distillation framework, where a powerful proprietary model (the teacher) generates training data for a smaller, more efficient model (the student).

The four-step pipeline for developing and deploying BURExtract-Llama, from raw report extraction to final structured inference.

The Four-Step Pipeline

  1. Preprocessing: Using regular expressions (regex), the system isolates the "Observations" and "Impressions" sections of the ultrasound reports, filtering out administrative metadata.
  2. Synthetic Labeling: A HIPAA-compliant version of GPT-4 was used to analyze reports and generate structured JSON labels. This process extracted 16 specific clinical keys, including lesion shape, orientation, and suspicion levels.
  3. Fine-Tuning: The researchers utilized Llama3-8B as the base model. It was fine-tuned on the GPT-4-generated labels using QLoRA (Quantized Low-Rank Adaptation), a technique that allows for high-performance training on standard hardware by updating only a small subset of model parameters.
  4. Inference: The resulting BURExtract-Llama model is deployed locally, allowing for rapid, private extraction of clinical concepts from new reports.
Detailed example of the structured JSON output generated by GPT-4, which serves as the gold-standard training label for the student model.

Data and Evaluation

The study utilized a dataset of 4,000 breast ultrasound reports from NYU Langone Health. The training and validation sets comprised 3,880 reports labeled automatically by GPT-4. To ensure a rigorous evaluation, a separate test set of 120 reports was meticulously labeled by human clinicians. This "gold standard" test set allowed the researchers to measure the model's performance against actual medical expertise rather than just its ability to mimic GPT-4.

Key Results

BURExtract-Llama demonstrated that a specialized 8-billion parameter model can achieve parity with much larger proprietary systems:

  • Performance Parity: The model achieved an average F1 score of 84.6%, nearly identical to GPT-4’s 84.7% and significantly higher than the base Llama-3 model (79.4%).
  • Perfect Formatting: The model achieved 100% "JSONable" accuracy, ensuring that the output is always in a valid format for direct database integration.
  • Efficiency: Using optimization libraries like vLLM, the model can process a report in approximately 2 seconds, making it suitable for real-time clinical workflows.
  • Domain Strengths: Interestingly, BURExtract-Llama outperformed GPT-4 in specific categories, such as identifying the anatomical region and depth of lesions.

Discussion and Limitations

The success of BURExtract-Llama highlights the viability of "Privacy-First AI" in healthcare. By specializing a smaller model for a narrow task, institutions can maintain data security without sacrificing accuracy.

However, the authors noted some limitations. Because the model was trained on GPT-4's outputs, it may inherit systematic biases or errors from the "teacher" model. Additionally, the dataset was sourced from a single hospital system; further external validation is required to ensure the model generalizes to the different reporting styles used at other institutions. Finally, the model occasionally struggled with "attribute confusion" in reports describing multiple distinct lesions, sometimes assigning a recommendation to the wrong finding.

Conclusion

BURExtract-Llama represents a significant step forward in clinical NLP. It proves that high-accuracy, structured data extraction is possible using open-source models that respect patient privacy and reduce institutional costs. Future work will focus on multi-center data integration and improving the model's ability to handle complex, multi-lesion reports.

Summary updated

Abstract

Breast ultrasound plays a pivotal role in detecting and diagnosing breast abnormalities. Radiology reports summarize key findings from these examinations, highlighting lesion characteristics and malignancy assessments. However, extracting this critical information is challenging due to the unstructured nature of radiology reports, which often exhibit varied linguistic styles and inconsistent formatting. While proprietary LLMs like GPT-4 effectively retrieve information, they are costly and raise privacy concerns when handling protected health information. This study presents a pipeline for developing an in-house LLM to extract clinical information from these reports. We first utilize GPT-4 to create a small subset of labeled data, then fine-tune a Llama3-8B using this dataset. Evaluated on a subset of reports annotated by clinicians, the proposed model achieves an average F1 score of 84.6%, which is on par with GPT-4. Our findings demonstrate that it is feasible to develop an in-house LLM that not only matches the performance of GPT-4 but also offers cost reductions and enhanced data privacy.
BibTeX
@inproceedings{chen2024burextract,
  author = {Yuxuan Chen and Haoyan Yang and Hengkai Pan and Fardeen Siddiqui and Antonio Verdone and Qingyang Zhang and Sumit Chopra and Chen Zhao and Yiqiu Shen},
  title = {{BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports}},
  booktitle = {MCHM@MM},
  year = {2024},
  url = {https://openreview.net/forum?id=DPgzS3cWwY},
}
Figure