Automatic clinical coding assigns a patient's discharge summary the full set of diagnosis codes it supports. It is a multi-label problem because patients usually have several conditions at once, and its leading models report very high ranking scores on public MIMIC data. Those scores cannot tell apart three kinds of model. The first actually reads which conditions the patient has. The second fires on disease words wherever they appear. The third has memorized document-to-label pairings from notes that circulate in training data. We propose to separate these with a single cheap tool: minimal edits that turn a factual mention of a diagnosis into a counterfactual one. A counterfactual is a statement about what would have happened but did not, as in "had the culture grown MRSA, vancomycin would have been continued." The edit leaves the disease words in place but removes the diagnosis from the correct answer.
A model that reads drops the code. A model that matches words keeps it. A model that memorized the note ignores the edit entirely. Pairing each counterfactual edit with a factual paraphrase, which changes the words but not the labels, gives a two-by-two design that attributes errors to each failure mode. Because every edit produces a new note-plus-label combination, the probe can be regenerated freshly for every evaluation and never becomes a static test set that leaks.
Core Claim
Label-changing counterfactual edits, contrasted with label-preserving factual paraphrases, measure how much of a clinical coder's apparent comorbidity understanding comes from memorization or lexical shortcuts rather than reading. Applying the same edits to documents a model was trained on versus documents it never saw gives a controlled, known-contamination condition. That condition lets us check that the probe actually detects memorization rather than assuming it does.
The secondary claim is a remedy. A counterfactual span detector adapted from general-domain work and placed in front of the coder should recover correct label sets on edited notes without hurting accuracy on unedited ones. Its value has to be shown against simpler fixes, discussed below.
Where This Sits
Three strands of work motivate the gap. The audit in Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models reviewed 53 medical benchmarks. It found 92% report no contamination check and over half reduce comorbid profiles to single answers, but it offers a rubric rather than a probe. ClinDEF avoids contamination by synthesizing new patient cases from medical knowledge graphs. It evaluates primary diagnosis in dialogue, not multi-label coding of real notes, and it does not touch counterfactual language. DyVal generates reasoning problems on the fly and showed static-benchmark leaders scoring near zero on them, but it works only on formal tasks such as arithmetic and graph reachability.
Our probe borrows DyVal's principle of regenerating items per evaluation. It differs from all three in one respect: it perturbs real clinical documents so that the correct label set changes, rather than synthesizing new cases or auditing old ones.
The closest mechanism is INVITE. INVITE distorts knowledge-graph facts into false-premise questions and found open models accept the false premise more than 80% of the time. INVITE tests whether a model challenges a false question. We instead test whether a coder correctly declines to assign a code when the note mentions the disease only in a contrary-to-fact clause. That is a different linguistic object and a different output space: thousands of codes rather than a free-text answer.
The counterfactual side supplies the method and constrains it. SemEval-2020 Task 5 defined sentence-level counterfactual recognition and antecedent/consequent span extraction on news text, including health journalism but no patient notes. IITK-RSA showed detectors fail on implicit counterfactuals marked by words like "wish" or "ought to have." ETHAN showed indicative and counterfactual conditionals share surface markers. These two negative results mean our edit generator must cover implicit and morphologically marked forms, not only "if… had… would have" templates. Otherwise a regular expression would trivially undo the probe. On the comorbidity side, Characterization of Co-occurring Pathologies in Clinical Notes argues that co-occurrence is the clinical norm. That is why we measure collateral damage to the other codes on an edited note, not just the edited code.
Research Program
The first component is the edit generator. For each MIMIC-IV discharge summary and each gold code, we locate supporting mentions by matching against code descriptions and synonyms, and by reading the label-aware attention of PLM-ICD. We restrict counterfactual edits to codes whose support is confined to identifiable mentions, so that removing the code is actually justified. We then rewrite every supporting mention into one of several clinically motivated counterfactual forms. The first is averted treatment ("would have required dialysis had creatinine continued to rise"). The second is hypothetical differential exclusion ("had this been typical asthma, symptoms would have resolved"). The third is speculative history. A fourth category consists of implicit forms without conditional markers, for example "the team wished they had started anticoagulation for the suspected embolism," where the embolism is not established.
Each counterfactual edit is paired with a factual paraphrase of the same mention. The paraphrase uses synonyms, abbreviations, or reordering and leaves the code in the gold set. A third variant inserts a counterfactual mention of a code absent from the note, deliberately sampled from the long tail where Mimic-IV-ICD reports that half of ICD-10 codes appear in three or fewer summaries. This variant tests whether rare-code predictions are triggered by mere mention.
Rewriting uses templates followed by language-model paraphrasing and an automatic consistency filter. A clinician-annotated sample of several hundred edits audits whether the gold label change is correct. The generator, not a fixed set, is the released artifact: each evaluation draws fresh documents, mentions, and surface forms from a seed.
The second component is the analysis design. For a coder $f$ and a note $d$ with gold code $c$, let $d^{cf}$ be the counterfactual edit and $d^{para}$ the factual paraphrase. We record whether $c$ survives in each. Reading predicts $c$ kept on $d^{para}$ and dropped on $d^{cf}$. Lexical matching predicts kept on $d^{cf}$ and possibly lost on $d^{para}$. Document memorization predicts kept on both, independent of what the edit says. We summarize with a counterfactual drop rate, a paraphrase retention rate, and the change in F1 on the untouched codes.
The known-contamination lever comes from running this design on two pools: documents from the coder's own training split and held-out test documents. A fine-tuned coder has, by construction, seen its training notes. If memorization drives edit-insensitivity, the drop rate should be lower on training documents than on test documents for fine-tuned models. It should not differ for a model that never saw MIMIC. If no such gap appears even under known exposure, the probe is not measuring memorization, and we will say so.
The third component is the remedy. We adapt a counterfactual span detector to clinical text and use it to mask or down-weight antecedent and consequent spans before coding. The detector is a cascaded design in the style of Yseop with the boundary label masking of Pheonix, trained on SemEval data and then on pseudo-labeled clinical sentences following HIT-SCIR. Span precision matters here. SemEval's best system reached 88.2% span F1 but only 57.5% exact match, and in this setting a span that is too wide deletes real diagnoses in neighboring clauses.
Baselines
The naive reference is the coder run unchanged on edited notes. That is the "do nothing" condition, and its drop rate is the quantity the probe reports. A second naive point is a coder that always predicts the original note's labels. It represents pure memorization and defines the floor where the drop rate is zero.
The strongest existing coder is PLM-ICD, which uses biomedical encoders, segment pooling for long notes, and per-code attention. It reports 59.8% micro-F1 on MIMIC-III (PLM-ICD). On full MIMIC-IV-ICD10, PLM-ICD and MSMN reach up to 97.07% macro AUC but 5.42% macro F1 (Mimic-IV-ICD). We retrain PLM-ICD with the published configuration on the Mimic-IV-ICD splits. All remedies are added on top of that same checkpoint, with matched tuning budgets on a shared validation set. We will also evaluate instruction-tuned language models prompted to code notes. No reported result found on Lacuna for language-model zero-shot coding on MIMIC-IV, so we establish that number ourselves with a fixed prompt budget.
For the remedy, the strongest competitor is counterfactual-augmented fine-tuning: training PLM-ICD on generated edits, an approach in the spirit of the syntactic perturbations in BLCU-NLP. It gets the same number of edited examples and the same training steps as the detector pipeline. Two further baselines complete the set. One is a regular-expression masker that removes sentences containing conditional and modal markers. The other is a negation and assertion filter that removes ruled-out mentions.
The ablation that isolates the key ingredient swaps the clinically adapted span detector for a sentence-level counterfactual classifier, which masks whole sentences. This tests whether span boundaries matter. The detector's own reference point is the SemEval best result of 90.9% sentence F1 (SemEval-2020 Task 5).
Beating these baselines means meeting four conditions together. The remedy must achieve a higher counterfactual drop rate on implicit edits than both augmentation and the regex masker. It must keep paraphrase retention and untouched-code micro-F1 within the validation noise band of unmodified PLM-ICD. And it must not raise false insertions of rare codes.
Why not just…?
Why not just mask sentences with "if," "had," or "would have"? This is the cheapest fix, and our edit generator is designed to defeat it on purpose. ETHAN and IITK-RSA show that such markers both over-fire on indicative conditionals and miss implicit counterfactuals. Clinical notes are full of indicative conditionals that carry real diagnoses, such as "if pain recurs, return." We report the regex masker separately for each edit category. If it matches the detector on everything, including implicit edits, the remedy component is unnecessary and the benchmark stands alone.
Why not just use existing negation and assertion detection? This is a real threat, because "ruled out" mentions overlap with hypothetical-differential counterfactuals. We include negation-only edits as a control category and run the assertion filter as a baseline. The proposal's claim survives only if averted-treatment and implicit counterfactuals, which contain no negation cue, break the assertion filter while the detector handles them. If they do not, we will report that counterfactual handling reduces to assertion detection in this domain.
Why not just fine-tune the coder on counterfactual-augmented notes? This may well work, and it is the main remedy competitor. Its risk is that it learns the generator's templates rather than counterfactual meaning. We test that directly by holding out entire counterfactual forms, such as implicit "wish" constructions, from augmentation and evaluating on them. If augmentation generalizes across held-out forms as well as the detector does, we would recommend augmentation as the cheaper fix. The probe remains the contribution either way.
Why not just evaluate on notes written after the model's training cutoff? Unveiling Causal Reasoning in Large Language Models used post-cutoff data to expose recall-driven reasoning, but public clinical corpora are frozen snapshots, and new notes are not released. Edits also do something cutoffs cannot: they flip a specific label while holding the rest of the document fixed. That lets us attribute errors to individual codes.
Proposed Experiments
The first experiment validates the probe. Clinicians label a stratified sample of edits for label correctness and clinical plausibility, and only edit types that pass are kept.
The second is the known-contamination check. We compare drop rates between training-split and test-split documents for fine-tuned PLM-ICD, for MSMN, and for a general language model that never saw MIMIC. This is the decisive test of whether edit-insensitivity tracks exposure. A significant training-versus-test gap only in the fine-tuned models supports the probe. No gap means the probe measures lexical shortcuts alone, which is still useful but narrower.
The third experiment reports the two-by-two decomposition for every coder. Results are broken down by code frequency band, which tests whether long-tail codes are more mention-triggered, and by counterfactual category.
The fourth evaluates the remedies under matched budgets. It reports drop rate, paraphrase retention, untouched-code micro-F1, rare-code false insertion, and the held-out-form generalization split.
The fifth tests freshness. We regenerate the probe with new seeds after models are fine-tuned on an earlier release, checking that scores obtained by training on one release do not transfer to the next. This is the property that makes the probe resistant to contamination rather than one more static set.
References
- Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
- ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- SemEval-2020 Task 5: Counterfactual Recognition
- IITK-RSA at SemEval-2020 Task 5: Detecting Counterfactuals
- ETHAN at SemEval-2020 Task 5: Modelling Causal Reasoning in Language Using Neuro-symbolic Cloud Computing
- Topic Modeling Based Classification of Clinical Reports
- PLM-ICD: Automatic ICD Coding with Pretrained Language Models
- Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification
- Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis
- Pheonix at SemEval-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling
- HIT-SCIR at SemEval-2020 Task 5: Training Pre-trained Language Model with Pseudo-labeling Data for Counterfactuals Detection
- BLCU-NLP at SemEval-2020 Task 5: Data Augmentation for Efficient Counterfactual Detecting
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?