Explanation Residue: Auditing What a Clinical Recommender Still "Remembers" After a Patient Revokes Data

When a patient uninstalls a connected health app or revokes a data source, the systems downstream usually delete the raw records and stop there. Forensic work shows how little that achieves. On a smart-scale ecosystem, deleting the app left the user's full history on the device and in the cloud (A Digital Forensics Investigation of a Smart Scale IoT Ecosystem). In LLM agents, deleting raw conversation logs left recoverable secrets in derived summaries in about 20% of cases, which was no better than doing nothing (Deployment-Time Memorization in Foundation-Model Agents). Clinical recommenders add a layer nobody has audited: the explanation. A system that suggests the next lab test and justifies it ("HbA1c has been stable for 18 months; cadence in the 90th percentile") is built from derived artifacts. These include cached feature summaries, cohort percentiles, and generated rationales, and each one can keep a revoked source alive after the ranking model itself has been fixed.

This proposal measures and removes that residue. We compare a clinical test recommender after a deletion with an "ideal world" copy rebuilt as if the revoked source had never been collected. The test is whether the two agree on what they recommend, and also on what they say and what their explanations depend on. Deletion also has a clinical consequence that makes the audit concrete. After revocation, the correct patient record is a fragmented one, so a compliant system should behave like one facing missing outside data: it should hedge, abstain, or re-order the test, not confidently cite a value it should no longer know.

Core Claim

Checking that a model's outputs match after deletion is not enough. Explanation layers built from derived features will keep citing values that come only from revoked data, even when the top-k recommendations already match the ideal-world system. We claim three things. First, an explanation-level audit detects residue that output-level checks miss. It combines value comparison with the cross-model feature-masking protocol of Beyond Ranking Accuracy, which replaces each feature an LLM cites with a median value and checks whether ranking quality drops on a separately trained model. Second, tagging every feature in an evidence card with the source records it was computed from lets a selective purge match a full rebuild at a fraction of the cost. Third, which artifacts must be purged, and which must be kept as audit tombstones, can be settled by a small Delphi panel. A Delphi panel is an anonymous, multi-round expert survey in which experts revise ratings after seeing peers' summarized responses, and consensus is defined statistically. Settling these questions this way, rather than letting the authors decide, matters because the panel's answers determine what "correct" post-deletion behaviour is.

The notion of correctness comes from Formalizing Data Deletion in the Context of the Right to be Forgotten. That paper defines deletion compliance as an observer being unable to tell the real system from one that never received the data. Its treatment is purely abstract. We turn it into an empirical test on recommendations and their rationales. For patient $p$ with revoked source $s$, let $E_D(p)$ be the set of features cited by the deleted system $D$, and let $v_D(f)$ and $v_I(f)$ be the values of feature $f$ in $D$ and in the ideal rebuild $I$. The explanation residue is

$$R(p,s)=\frac{1}{|E_D(p)|}\sum_{f\in E_D(p)}\mathbf{1}\left[v_D(f)\neq v_I(f)\ \text{or}\ f\notin \text{features}(I)\right],$$

that is, the fraction of cited evidence that the ideal world could not have produced. We pair it with an influence residue: the change in ranking score when lineage-tagged features are masked in $D$. That change should be zero in a compliant system, because those features should already carry no information.

How This Differs From the Closest Work

Two papers connect deletion and explanation, and both motivate this work rather than doing it. Verifying Machine Unlearning with Explainable AI uses saliency maps to check that an unlearned vision model no longer attends to a forgotten class. It audits model weights on images. We audit cached explanation artifacts in tabular and sequential clinical recommendation, where the residue can sit entirely outside the model. Deployment-Time Memorization in Foundation-Model Agents finds the summary-tier residue and shows that full-pipeline purge removes it. It uses verbatim high-entropy canaries on 50 synthetic chat users. Clinical residue is mostly numeric influence, such as a trend or a percentile, which canary extraction cannot detect. We therefore need an influence test, not an extraction test.

On the clinical side, three papers supply the method and expose the gap. DiaLLMs turns LOINC and ICD-10 codes into demographic-adjusted sentences for test recommendation. RAM-EHR summarizes retrieved knowledge into sentences for EHR prediction. Both create exactly the kind of derived text that escapes deletion, and both assume patient history is permanently available. The "norm-prompt" result of Beyond Ranking Accuracy is especially relevant: framing features as user-relative percentiles improved LLM attribution. Cohort percentiles, however, are computed over other patients, so one patient's revocation silently shifts everyone else's explanations. No existing work measures this cross-patient residue.

CliniCARE-Bench constrains the design. It shows that agents guess instead of abstaining on fragmented records, and that about one in five correct verdicts rests on a forbidden shortcut such as an unconfirmed billing code. Post-deletion records are fragmented by construction, so we expect the same failure. Its MUST-DO and MUST-NOT rubrics came from a single internal clinical board. We instead use a Delphi process with Content Validity Index scoring, following instrument-development practice (Wake et al.; Vu et al.), to label acceptable post-deletion behaviour.

Research Program

Pipeline. We build a deliberately conventional two-part recommender, following the finding in Beyond Ranking Accuracy that gradient-boosted trees rank better than LLMs while LLMs explain well. A gradient-boosted ranker recommends the next laboratory test from longitudinal MIMIC-IV records. An LLM explanation layer reads an evidence card of interpretable features covering lab cadence, recency, trend, cohort percentile, and DiaLLMs-style code-to-text statements, and writes a rationale citing specific card entries. Cards, cohort norms, and generated rationales are cached, as they would be in deployment.

Simulating revocation. We split each patient's history into simulated "sources" that mirror real fragmentation: outpatient versus inpatient labs, and a device-like stream of vital signs. Revoking a source deletes its records. Every card feature carries lineage tags naming the source records it was computed from.

Deletion policies. We compare policies of increasing depth. The first deletes raw records only. The second also retrains the ranker. The third additionally regenerates the affected patient's cards and rationales. The fourth, lineage-guided purge, also recomputes every cohort norm and every other patient's card that touches the revoked lineage. The ideal-world full rebuild is the reference.

Agent setting. We repeat the analysis in MedAgentBench, whose FHIR environment allows resources to be deleted through the API. This tests whether agent-held memory and tool caches show the same residue.

Delphi panel. A two- to three-round modified Delphi with roughly 15 to 20 clinicians and health-data-governance experts rates two kinds of items. The first are artifacts, such as cached rationales, cohort norms, and audit logs, to be classed as must-purge, keep-as-tombstone, or keep. This is the GDPR-versus-retention tension raised by BRIDGE. The second are post-deletion behaviours for sampled cases, classed as recommend, hedge, re-order the test, or abstain. Items reaching consensus under a prespecified Content Validity Index threshold become evaluation labels. Items that do not reach consensus are reported as genuinely contested, not forced into a label.

Baselines

The do-nothing reference leaves all artifacts untouched after revocation. It sets the upper bound on residue.

The raw-only deletion baseline mirrors what Deployment-Time Memorization in Foundation-Model Agents found to be statistically indistinguishable from doing nothing in its summary tier, where it left residue in about 20% of cases. It is the policy most real systems implement, so it is the main practical competitor.

The retrain-ranker baseline is the strongest output-level method. It recomputes the model exactly, so any residue it leaves lives only in the explanation layer. This makes it the cleanest test of our central claim.

The full-pipeline purge or ideal-world rebuild is the strongest existing method. Chen et al. report that it drives summary residue to zero. It is our correctness reference, not a competitor to beat on residue. Lineage-guided purge beats it only if it matches it, with residue indistinguishable from zero under a bootstrap test, at substantially lower recompute cost measured in features and rationales regenerated and wall-clock time.

All policies run on the same patient splits, the same revocation events, the same ranker hyperparameters tuned once on non-revoked data, and the same LLM and prompts with fixed decoding.

On ranking utility, the closest reported reference is DiaLLMs' Recall@5 of 79.54% on TriNetX-Metabolic (DiaLLMs). We cannot assume access to TriNetX, so this number frames scale rather than serving as a head-to-head target. For explanation fidelity, the reference is the 99.6% numeric consistency reported on Instacart (Beyond Ranking Accuracy). For residue on clinical recommendation explanations, no reported result was found on Lacuna, so the baselines' numbers will be produced here.

The key ablation removes lineage tags and purges by record ID only. It isolates whether provenance, rather than simply more recomputation, removes residue. A second ablation keeps lineage but skips cohort-norm recomputation, to measure cross-patient residue on its own.

Why not just…?

Why not just rebuild everything from scratch on every revocation? Rebuilding is correct by definition, and for a tree ranker retraining is cheap, so this is a real threat to the efficiency half of the claim. The cost lies elsewhere. Cohort percentiles mean that one revocation changes other patients' cards, and regenerating every LLM rationale for a large cohort on each request is expensive. Rebuilding also destroys audit trails that the Delphi panel may require as tombstones. If our cost measurements show that full rebuilds are affordable at realistic revocation rates, we will concede the efficiency point. The audit metric would still stand as the way to verify a deployment that claims to rebuild.

Why not just check that recommendations match the ideal world? That check is the retrain-ranker baseline. Our core hypothesis is that it passes while explanations still cite revoked values. The decisive experiment reports both quantities on the same revocations. If explanation residue is non-zero only when recommendations also diverge, the explanation-level audit adds nothing and the claim fails.

Why not just use canary extraction or membership inference, as Chen et al. do? Canaries detect verbatim secrets. Clinical residue is mostly a shifted trend or percentile with no unique string to extract. We will run canary extraction alongside our metrics, planting synthetic marker values in revoked sources, and report how much residue it misses relative to value comparison and lineage masking.

Why not let the authors or an internal board define acceptable post-deletion behaviour? That is what CliniCARE-Bench did, and it is cheaper. We test whether the Delphi process changes the answer by having the authors pre-register their own labels before the panel runs. If agreement with the Delphi consensus is near-perfect, we report that Delphi was unnecessary for this question.

Proposed Experiments and Decisive Checks

The main experiment revokes sources for a held-out set of MIMIC-IV patients. For each deletion policy we report Recall@5 and top-k overlap with the ideal world, explanation residue $R$, influence residue under lineage masking, cross-patient residue in non-revoking patients, and recompute cost. A replication on Instacart, using the evidence-card protocol of Beyond Ranking Accuracy with simulated account deletions, checks that the effect is not specific to clinical data. The MedAgentBench run measures whether agents, after FHIR resource deletion, still cite deleted observations. It reports task success alongside residue, against the reported 69.67% overall success of Claude 3.5 Sonnet v2 (MedAgentBench).

The behavioural check scores each policy's post-deletion outputs against the Delphi labels. In particular, it measures how often systems confidently recommend from evidence the panel deems no longer available, which is the under-abstention failure that CliniCARE-Bench documented, now induced by deletion.

Three outcomes would decide the matter. The claim is supported if retrain-ranker shows near-zero output divergence but clearly non-zero explanation residue. The proposed method works if lineage-guided purge matches the full rebuild on residue at lower cost. Panel-defined labels matter if they reorder the policies relative to the authors' pre-registered labels. Each of these can fail, and each failure would be reported as a finding.