Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation

Pablo Messina, René Vidal, Denis Parra, Álvaro Soto, Vladimir Araujo

Abstract

Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from free-text radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a Fact Extractor that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a Fact Encoder (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature.

Introduction

Radiology reports are a critical component of clinical care, yet their unstructured, free-text nature makes them difficult for machine learning models to parse effectively. These reports often contain "noise"—typos, verbose descriptions, and complex sentence structures—that obscure the underlying clinical facts.

Existing text encoders frequently fail to capture essential clinical nuances. For instance, a standard model might treat "pneumonia is present" and "no evidence of pneumonia" as semantically similar due to high word overlap, despite their opposite clinical meanings. Furthermore, traditional evaluation metrics like BLEU or ROUGE rely on token-level overlap, which is insufficient for medical contexts where a single word (e.g., "no" or "stable") can fundamentally alter a diagnosis. This research addresses these gaps by shifting the focus from word-level patterns to factual representation.

Methodology

Fact Extraction

The researchers propose a "teacher-student" approach to extract concise factual statements from messy reports. Instead of rigid, rule-based systems, they leverage the reasoning capabilities of LLMs like GPT-4 and ChatGPT to identify facts. To scale this process cost-effectively across millions of reports, they use knowledge distillation: the LLM labels a subset of data, which is then used to fine-tune a smaller, more efficient T5-small model. This "student" model can then process the entire dataset with high fidelity to the LLM's performance.

The two-stage Fact Extraction process: using LLMs for initial labeling and distilling that knowledge into a T5 model.

Fact Encoding (CXRFE)

The second stage involves a specialized BERT model, CXRFE, fine-tuned using a multi-task learning protocol. The training incorporates several key objectives:

  • Triplet Loss: Teaching the model to group clinically similar facts together while separating dissimilar ones.
  • Natural Language Inference (NLI): Training the model on the logic of entailment and contradiction to ensure embeddings reflect clinical truth.
  • Sentence Decoding: An autoencoder task that ensures the embeddings retain sufficient information to reconstruct the original factual statement.

CXRFEScore

To improve the evaluation of AI-generated radiology reports, the authors introduced CXRFEScore. Unlike traditional metrics, CXRFEScore uses the CXRFE model to break both a "reference" and a "candidate" report into atomic facts. These facts are embedded and compared using cosine similarity, allowing the metric to recognize factual accuracy even when the wording differs.

The CXRFEScore pipeline: decomposing reports into facts, embedding them with CXRFE, and calculating similarity.

Results and Discussion

The framework was evaluated against massive datasets including MIMIC-CXR, Chest ImaGenome, and RadNLI. Key findings include:

  • Superior Ranking: CXRFE outperformed state-of-the-art models like BioViL-T in triplet ranking, particularly on "hard triplets" designed to test subtle clinical contradictions.
  • State-of-the-Art NLI: The model achieved 89.8% accuracy on the RadNLI test set, surpassing GPT-4 with Chain-of-Thought prompting (89.0%).
  • Metric Reliability: CXRFEScore demonstrated a significantly higher correlation with radiologist-informed standards than BLEU or ROUGE. It proved far more effective at penalizing reports with clinical contradictions, which traditional metrics often overlook.

Conclusion

By prioritizing factual statements over individual tokens and leveraging LLM-based distillation, this framework significantly advances radiological text representation. Future work will aim to address current limitations in sentence tokenization and expand the scope to include patient history and multimodal data (integrating X-ray images).

Summary updated

Abstract

Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from free-text radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a \textit{Fact Extractor} that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a \textit{Fact Encoder} (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at \url{https://github.com/PabloMessina/CXR-Fact-Encoder}.
BibTeX
@inproceedings{messina2024extracting,
  author = {Pablo Messina and René Vidal and Denis Parra and Álvaro Soto and Vladimir Araujo},
  title = {{Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.236},
}
Figure