Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation

Pablo Messina, René Vidal, Denis Parra, Alvaro Soto, Vladimir Araujo

Introduction

Radiology reports are essential clinical documents, yet their "free-text" nature presents a significant challenge for automated processing. These reports are often verbose, contain typos, and use complex linguistic structures to describe clinical findings. Standard text encoders, such as BERT, frequently struggle in the medical domain, particularly with distinguishing between paraphrases and critical contradictions (e.g., "pneumonia" versus "no pneumonia"). In a diagnostic setting, failing to recognize a negation is not merely a linguistic error but a potential clinical failure. This research introduces CXRFE, a specialized medical text encoder designed to represent radiological findings with high factual precision and robustness.

The CXRFE Framework

The researchers developed a two-stage framework to transform noisy clinical text into structured, meaningful representations:

  1. Fact Extraction: To bypass the noise of raw sentences, Large Language Models (LLMs) like GPT-4 are used to decompose reports into "atomic facts"—short, unambiguous clinical statements. To ensure scalability and cost-effectiveness, this knowledge is distilled into a smaller T5-small model trained to replicate the LLM's extraction performance.
  2. Fact Encoding (CXRFE): Building upon CXR-BERT, the authors implemented a multi-task learning protocol. The model is trained on six simultaneous tasks, including Triplet Loss for semantic proximity, Natural Language Inference (NLI) for logical consistency, and classification for anatomical mapping.

A New Metric: CXRFEScore

Traditional metrics like BLEU or ROUGE, which rely on word overlap, are often misleading in medical contexts. To address this, the authors proposed CXRFEScore. This metric extracts facts from both a generated and a reference report, encodes them using CXRFE, and calculates similarity within a high-dimensional latent space.

The CXRFEScore pipeline: from report tokenization and fact extraction to similarity matrix calculation.

Key Results and Performance

The CXRFE model demonstrates superior performance across several benchmarks, including MIMIC-CXR and RadNLI. A standout finding is the model's ability to handle medical logic and contradictions. By explicitly training on NLI tasks, CXRFE learns that "pleural effusion" and "no pleural effusion" must be distant in vector space, despite sharing most tokens.

In head-to-head evaluations for NLI accuracy, CXRFE achieved 89.8%, surpassing even GPT-4 with Chain-of-Thought (CoT) prompting. Furthermore, CXRFEScore showed a significantly higher correlation with radiologist-annotated labels compared to traditional metrics like BERTScore or RadGraph F1.

Comparison of NLI accuracy across various models, highlighting CXRFE's superior performance against state-of-the-art LLMs.

Conclusion

By shifting the focus from raw sentences to atomic facts and leveraging LLMs as "teachers" for smaller, specialized encoders, this work provides a robust path forward for medical AI. CXRFE not only improves text understanding but also introduces a more clinically relevant way to evaluate generated medical reports through CXRFEScore. Future work aims to integrate these high-quality text representations into multimodal vision-language models for comprehensive radiological analysis.

Summary updated

Abstract

Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from free-text radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a \textit{Fact Extractor} that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a \textit{Fact Encoder} (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at \url{https://github.com/PabloMessina/CXR-Fact-Encoder}.
BibTeX
@inproceedings{messina2024extracting,
  author = {Pablo Messina and René Vidal and Denis Parra and Alvaro Soto and Vladimir Araujo},
  title = {{Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.236},
}
Figure