Privacy-Preserving Retrieval for Auditable Clinical Language Modeling on Real-World Radiology Data

Nuri Purswani, Viktor Schlegel, Anil Anthony BharathResearch Square 2026 · DOI

TL;DR

Researchers have developed a retrieval-based framework (R-LM) that allows Large Language Models to perform clinical tasks without "baking" sensitive patient data into their permanent weights. By keeping medical records in an external, auditable datastore and combining predictions at the token level, the system recovers up to 67% of the performance gains of fine-tuning while remaining fully compliant with data privacy laws like GDPR.

What Problem Does This Paper Address?

When we want an LLM to understand specialized domains like radiology, the standard approach is Supervised Fine-Tuning (SFT). We feed the model thousands of private medical records, and it adjusts its internal weights to learn the patterns.

However, in healthcare, this creates a "black box" privacy nightmare. Once patient data is absorbed into model weights, it is nearly impossible to remove. This conflicts with the "right to erasure" (GDPR), as you can't simply "un-learn" a specific patient's data without retraining the entire model. Furthermore, LLMs are susceptible to extraction attacks where clever prompting can trick the model into leaking the sensitive data it was trained on.

The authors address this by asking: Can we get the performance of a specialized clinical model while keeping the patient data completely separate from the model itself?

Methods

The authors evaluate a Retrieval-based Language Model (R-LM) framework. Unlike standard Retrieval-Augmented Generation (RAG), which stuffs retrieved text into a prompt, R-LMs work at the token probability level:

  1. Frozen Base Models: They used three "off-the-shelf" 1-billion parameter models (LLaMA-3.2, Qwen-2.5, and OLMo-2) that had never seen the private clinical data.
  2. External Datastore: They built a vector database (using FAISS) containing millions of "key-value" pairs from radiology reports. The "key" is a mathematical representation of a clinical context, and the "value" is the actual next token that followed in the real-world report.
  3. Token Interpolation: During generation, the system calculates two probability distributions for the next word: one from the base LLM and one from the external datastore.
  4. Retrieval Strength ($\lambda$): A tunable parameter combines these two. For example, a 50% retrieval strength means the final prediction is an equal mix of the "general" LLM's logic and the "specific" datastore's evidence.

Datasets

The study utilized the MIMIC-IV v2.2 database, a massive, de-identified collection of electronic health records. Specifically, they focused on radiology reports, using the "Findings" section as input and the radiologist-authored "Impression" (summary) as the ground-truth target. The evaluation was performed on a blind test set of 500 reports.

Key Insights and Ideas

  • The "Right to be Forgotten" is Now Possible: Because the clinical data lives in an external database rather than the model's "brain," administrators can audit, update, or delete specific patient records instantly without touching the LLM.
  • The 50% Sweet Spot: The researchers found that performance peaks at intermediate retrieval strengths (around 50%). If you rely too much on the datastore (e.g., 90%), you suppress the base model’s linguistic reasoning, leading to unstable or nonsensical outputs.
  • Efficiency vs. Privacy: While full fine-tuning is still the "gold standard" for accuracy, R-LMs provide a "middle way" that is legally safer and computationally cheaper than constant retraining.

Key Results

The R-LM approach showed significant improvements over generic base models across all tested architectures:

  • Performance Recovery: R-LMs recovered 32–38% of the ROUGE-L (structural overlap) gains and 49–67% of the perplexity (fluency/confidence) gains achieved by full fine-tuning.
  • Statistical Significance: All improvements were statistically significant ($p < 0.0001$).
  • Model Agnostic: The benefits were consistent across LLaMA, Qwen, and OLMo, suggesting this is a robust architectural choice rather than a fluke of one specific model.

Limitations and Future Work

  • The Hallucination Hurdle: While retrieval makes the models more "confident" and fluent, it doesn't automatically stop them from hallucinating (stating clinical facts that aren't in the source text). The authors note that hallucination mitigation remains a critical future focus.
  • The Performance Gap: There is still a "performance tax" for privacy. Fine-tuned models still outperform R-LMs, meaning more work is needed on how we represent and retrieve clinical "keys" to close that remaining 30-60% gap.
  • Reasoning Limits: Previous research suggests that token-level retrieval can struggle with complex, multi-step clinical reasoning compared to models that have internalized the data.

Summary updated

Abstract

<title>Abstract</title> Clinical large language models adapted for real-world use are commonly fine-tuned on patient data, embedding confidential information within model parameters and limiting auditability and privacy. We evaluate a retrieval-based framework separating clinical data from the language model by storing patient records in externally governed memory. On a radiology report summarisation task, retrieval recovers 32–67% of fine-tuning gains (perplexity, ROUGE-L; <italic>p</italic> &lt; 0.0001), while hallucination reduction remains a future focus.
BibTeX
@inproceedings{purswani2026privacy,
  author = {Nuri Purswani and Viktor Schlegel and Anil Anthony Bharath},
  title = {{Privacy-Preserving Retrieval for Auditable Clinical Language Modeling on Real-World Radiology Data}},
  booktitle = {Research Square},
  year = {2026},
  doi = {10.21203/rs.3.rs-8745792/v1},
}
Figure