When Withholding a Record Isn't Enough: Measuring and Closing Revocation Leakage in Clinical Retrieval

Suppose a patient tells the hospital's AI assistant to stop using the record of a past psychiatric admission or an HIV prescription. The usual engineering answer is to delete that record from the search index the assistant reads from. In systems that chain several lookups together to answer a question, deletion causes two failures, and they pull in opposite directions. The first is that the withheld fact often survives in the records that remain: an antiretroviral in a later medication list, a follow-up note, a billing code. A reader model can reconstruct what was hidden. The second is that silent deletion looks like clinical absence, so the assistant may confidently write "no psychiatric history," which is a false negative with clinical consequences.

We propose to measure the first failure precisely and to fix it with a procedure we call consent closure. When a revocation arrives, closure finds the smallest additional set of surviving record spans whose removal makes "this patient withheld X" statistically indistinguishable from "this patient never had X," judged by an attacker model. The second failure is handled by a uniform marker attached to every patient, which tells the reader that records may be incomplete. Because every patient carries the marker, its presence reveals nothing. The work is a benchmark, an attack, and a defense, all on real multi-admission clinical records.

Core Claim

For multi-hop clinical retrieval, the privacy promise a patient cares about is not "the record is gone from the index." It is "an observer of the system's view of me cannot tell whether I have the withheld condition." We claim three things. First, delete-only revocation leaves a large, measurable gap between those two promises, because clinical records are densely cross-referenced. Second, that gap can be closed per patient with far less collateral redaction than blanket category filtering. Third, closure plus a uniform incompleteness marker preserves question-answering accuracy on questions that never depended on the withheld record.

We define leakage as an attacker's advantage at telling two groups apart. The first group is patients who had fact $f$ and revoked it. The second is matched patients who never had $f$. The attacker sees only each patient's post-revocation view $V$:

$$\mathrm{Adv}(f) = \mathrm{AUC}\big(A(V) : \text{withheld } f \ \text{vs.}\ \text{never had } f\big) - 0.5,$$

where $A$ is any attacker model that outputs a score from $V$. An $\mathrm{Adv}$ near zero is the target.

Where This Sits

Three strands of prior work motivate this proposal.

Privacy-Preserving Retrieval for Auditable Clinical Language Modeling on Real-World Radiology Data keeps MIMIC-IV radiology text in an external datastore read by frozen 1B models, and argues erasure becomes an $O(1)$ index deletion. We take that deletion as our starting baseline and ask whether it achieves erasure of the fact rather than of the row. That paper evaluates single-turn Findings-to-Impression generation, where cross-record reconstruction cannot arise.

When Machine Unlearning Meets Retrieval-Augmented Generation simulates forgetting by retrieving a confidentiality instruction for the forgotten concept. It reports near-100% unlearning success on open-domain data, falling to about 20% under adaptive attacks. Its guard fires only when the forgotten concept itself is retrieved. It does nothing about a surviving medication list that implies the concept.

The multi-hop retrieval literature supplies the mechanism behind the problem. HopRetriever and FrugalRAG show that answers are assembled from bridge facts across documents, and both assume the full corpus is available. We use exactly that bridging property: the chains that let a reader answer questions also let it reconstruct withheld nodes.

On the access-control side, the self-sovereign identity systems MediLinker and PrEPLinker give patients revocation rights over stigmatized records. They treat revocation as a deterministic access decision with no model of what downstream reasoners infer. We take their revocation events as input and do not build cryptography.

E-Tenon constrains the design. It reports about 761 ms per attribute-based decryption, which is too expensive to repeat at every retrieval hop. So closure runs once, at revocation time, and produces an ordinary redacted view that queries can read with no per-hop cryptographic cost.

The Janus Interface shows that fine-tuning on 10–30 examples revives suppressed personal-data associations. It shapes our threat model: attackers may be fine-tuned, and we separately test whether the base readers carry memorized clinical text.

Research Program

Building the benchmark. We construct a revocation benchmark from MIMIC-IV using two resources:

  • EHRNoteQA (EHRNoteQA): 962 clinician-verified questions over multi-admission discharge summaries.
  • EHRXQA (EHRXQA): structured tables where prescriptions, labs, and diagnoses link across admissions.

A revocation withholds every admission, note, or table row that directly records a target fact $f$. Targets come from stigmatized categories: HIV, substance use disorder, psychiatric diagnoses, and reproductive care.

Each withheld patient is paired with a control patient matched on age, sex, admission count, and unrelated comorbidities who never had $f$. Questions are labeled either as depending on the withheld record or as independent of it. The labels are derived from the evidence annotations and SQL provenance, with clinician spot-checks.

Attacker suite. We use attackers of increasing strength:

  • a gradient-boosted model on structured codes;
  • a fine-tuned clinical encoder over surviving text;
  • an instruction-tuned 7B reader prompted to infer the hidden condition;
  • that same reader fine-tuned Janus-style on 30 labeled revocations.

Consent closure. A lightweight probe attacker is trained on synthetic revocations over training patients. For a new revocation, the procedure repeats two steps until the probe's score for that patient falls inside the score distribution of matched controls:

  1. Attribute the probe's score to individual surviving spans (medication entries, code rows, sentences).
  2. Remove the highest-attribution span.

The output is a list of proposed extra redactions, which can be shown to the patient for approval. Separately, every patient's retrieved context carries the same fixed marker stating that some records may be restricted. The reader is tuned on a small set of examples so that it answers "not documented in available records" rather than asserting absence.

Guarding against overfitting to the probe. Closure is always evaluated against attackers it did not train against, including the fine-tuned 7B reader.

Baselines

The no-revocation reference answers questions over the full record. It gives the utility ceiling and a leakage value that is maximal by construction. For context, EHRNoteQA reports GPT-4 at 97.16% multi-choice and 91.30% open-ended, and Llama3-70b-Instruct at 94.33% multi-choice (EHRNoteQA). EHRXQA reports 92.9% execution accuracy on table questions with its NeuralSQL pipeline (EHRXQA). We re-run our own 7B readers under identical prompts rather than borrowing these numbers.

The naive baseline is delete-only: the Purswani et al. approach of removing the directly revoked records and nothing else. It is the default practice and costs nothing in utility on independent questions.

The strongest existing privacy method is behavioral confidentiality prompting (When Machine Unlearning Meets RAG). We inject a confidentiality instruction keyed to the revoked concept into the same index and the same reader. That paper's reported figures are near-100% unlearning success on generic tasks and about 20% under adaptive exposure. No reported result was found on Lacuna for inferential leakage or for any clinical revocation setting, so we will establish the number ourselves.

The strongest simple redaction competitor is category redaction. A classifier removes every span semantically related to the revoked category, such as all antiretrovirals and all infectious-disease notes.

Two ablations isolate the ingredients:

  • Closure without the marker tests whether false-negative assertions come back.
  • Marker without closure tests whether the marker alone does anything for leakage.

All methods share the reader models, retriever, index, prompt budget, and a matched tuning budget, and all are evaluated on held-out patients.

Closure "beats" the competitors in either of two ways:

  • At equal attacker advantage (within bootstrap confidence intervals), it retains significantly more accuracy on independent questions than category redaction and suppresses fewer spans.
  • It reaches an advantage below 0.05 where delete-only and confidentiality prompting do not.

Why not just…?

Why not just delete the revoked records, since the index is the only place the data lives? Deletion removes the row, not the implication. Nothing in the file measures how much clinical records imply each other, and our first experiment is the go/no-go test. If delete-only already yields attacker advantage near zero on MIMIC-IV, the leakage half of this proposal is unnecessary, and we would report that as the finding.

Why not just use confidentiality prompts, which already achieve near-perfect unlearning? Those prompts trigger on retrieval of the forgotten concept. Here, the dangerous evidence is a different record: a drug name, a lab, a clinic code. The retriever surfaces it for unrelated reasons, and the reader draws the conclusion itself. The same paper reports success falling to about 20% under adaptive probing. Our fine-tuned attacker tests that regime directly, with the confidentiality baseline run on identical inputs.

Why not just redact the whole sensitive category with a classifier? This is the real threat to the proposal. If a good category filter already drives advantage to zero at acceptable utility cost, per-patient closure adds complexity for nothing. The headline comparison settles it: independent-question accuracy and spans removed, measured at matched advantage. We expect category redaction to over-remove. For example, stripping every antiviral also removes hepatitis B treatment for patients who revoked HIV. But this is an empirical claim we could lose.

Why not just tell the model the records may be incomplete? That is exactly our marker. We expect it to fix the false-negative side and do nothing for leakage, and the marker-only ablation tests this. The marker is applied uniformly because a marker shown only to revoking patients would itself reveal that a revocation exists. We measure that effect with a variant attached only to revokers.

Proposed Experiments and Decisive Checks

We will run four experiments, in this order.

1. Go/no-go leakage measurement. We measure delete-only attacker advantage per target category on EHRNoteQA and EHRXQA patients. If no category shows advantage above 0.1 even for the fine-tuned 7B attacker, we stop the leakage line and publish the negative result.

2. Head-to-head comparison. Assuming leakage exists, we compare closure against category redaction, confidentiality prompting, and delete-only on three measures:

  • attacker advantage against held-out attackers;
  • accuracy on dependent and independent questions;
  • the rate of unsupported absence assertions, judged by clinicians on a 200-answer sample.

3. Parametric memory check. We test whether the frozen readers have memorized MIMIC text, using prefix-completion extraction on notes held out from the index. If they have, closure cannot close the parametric channel. In that case we report advantage separately for a reader known not to have seen MIMIC.

4. Revocation-time cost. We measure how long closure takes when a revocation arrives. It must fit within a clinic's consent workflow, and it adds no per-query latency.

The proposal succeeds if closure matches category redaction's leakage while clearly retaining more utility. It fails informatively if either delete-only proves already safe or category redaction proves already sufficient.