Patients usually read at a 6th to 8th grade level, while their discharge notes, radiology reports and systematic reviews are written for clinicians. Automatic "clinical simplification" systems rewrite that text into plain language. They are tested only on clean, carefully formatted inputs. Real clinical notes are messy. The same drug may appear under a brand name or a generic name, doses are written as "10mg" or "ten milligrams", abbreviations vary from clinic to clinic, and typos are common. The failure that matters is not a slightly worse readability score. It is a plain-language summary that silently changes a dose, drops a negation, or loses a diagnosis because the input was phrased a little differently. We propose to measure this failure and to fix it with adversarial training cheap enough to run routinely. Adversarial training means training on deliberately worst-case versions of the inputs.
The core idea has three parts. First, restrict perturbations to ones that keep the clinical meaning, using links to medical vocabularies (ontologies), so the reference plain-language target stays valid. Second, choose the perturbation that most threatens the clinically critical words of the output (numbers, units, drug and disease names, negations), rather than the one that most damages the output as a whole. Third, find that perturbation with a single gradient computation against the reference text, instead of decoding candidate outputs over and over. If this works, a simplifier can be hardened against realistic record noise for a small multiple of normal fine-tuning cost, and the hardening targets the errors that would actually hurt a patient.
Core Claim
Our central hypothesis is that, at matched training compute, fact-targeted gradient-selected perturbations make a sequence-to-sequence clinical simplifier markedly more consistent on critical facts under realistic input variation than random noise augmentation or a normalization preprocessor does. We also expect this to cost little or nothing in readability or clean-input quality. A secondary hypothesis concerns cost: the approximate attack should recover most of the robustness gain of an expensive search that decodes every candidate, at a fraction of the GPU time.
Two neighbouring results make this concrete.
The first motivates the problem. BBAEG builds a black-box attack on biomedical classifiers that works in stages. It first swaps medical entities for synonyms linked through MeSH and DrugBank, then adds typographical noise, then rewrites numbers as words. Perturbing about 8% of words drops SciBERT accuracy on the adverse-drug-event corpus from 0.88 to 0.50. That work attacks classifiers and proposes no defense. We reuse its meaning-preserving perturbation families, apply them to generation, and add a defense.
The second supplies the method. A2T makes text adversarial training affordable. It ranks words by the gradient of the loss with respect to their embeddings instead of deleting words one at a time, uses a distilled similarity model, and attacks only 20% of each epoch's data. It reports roughly a 3x speedup over TextFooler, but only on classification, and it explicitly leaves generation as future work. It also reports that its masked-language-model substitution variant hurt clean accuracy through semantic drift. Our ontology constraint is designed to avoid that drift.
The closest work that combines simplification with adversarial ideas is LAE-LS. There, the adversary is a style discriminator that teaches a general-domain lexical simplifier which words to mask. LAE-LS does not study robustness to perturbed inputs and never touches clinical text. Our adversary perturbs inputs, and our goal is robustness, not style transfer.
Research Program
The perturbation space $S(x)$ for a source text $x$ is built at the positions that BBAEG-style entity linking marks as clinical entities, numbers or abbreviations. Each position gets only meaning-preserving substitutes:
- ontology synonyms and brand/generic swaps,
- digit/word and unit-format rewrites,
- expansion or contraction of abbreviations,
- character-level typos on non-numeric tokens.
Negations and numeric values themselves are never altered. Because $x' \in S(x)$ keeps the meaning of $x$, the reference simplification $y$ remains a correct target.
The training loss upweights the critical spans of the target:
$$L_{\text{fact}}(\theta; x', y) = \sum_t w_t \big(-\log p_\theta(y_t \mid y_{<t}, x')\big), \qquad w_t = \lambda \text{ if } y_t \text{ lies in a critical span, else } 1.$$
Here $p_\theta$ is the simplifier's next-token probability with parameters $\theta$, $y_t$ is the $t$-th reference token, and $\lambda > 1$ controls how strongly numbers, units, entities and negations are weighted.
The adversary does not decode. For each candidate site $i$ and substitute $s$, it scores the first-order change
$$\Delta_{i,s} \approx (e_s - e_{x_i})^\top \nabla_{e_{x_i}} L_{\text{fact}},$$
where $e$ denotes the token embedding and the gradient comes from one teacher-forced forward and backward pass. Multi-token substitutes use the mean of their embeddings. The adversary then applies the top-$k$ substitutions and runs one confirming forward pass. The approximation is justified because the perturbations are small in embedding space, which is the same reasoning behind A2T's saliency ranking. The total cost is about three passes per attacked example. A search that decodes each candidate costs one full generation per candidate.
We will train BART-base simplifiers in two regimes:
- full fine-tuning;
- a frozen backbone with a small set of learned prefix vectors.
The second regime transfers Robustness Tokens to text. That paper adds 10 learnable tokens to frozen vision transformers and restores DINOv2-G white-box accuracy from 0.3% to 63.1% within 200 steps. It attributes the effect to suppressing outlier activations. Whether that effect exists in encoder-decoder language models is open, so we treat the prefix regime as a cost arm and do not assume it will work.
We will also test whether combining our method with the unlikelihood jargon penalty of Devaraj et al. keeps its readability gains. That penalty lowered Cochrane Flesch-Kincaid grade from 13.44 for BART to 11.97, but the paper noted hallucinated boilerplate.
Evaluation Setting
We use two public corpora from the file. MedLane is sentence-level clinical simplification derived from MIMIC-III, with 12,801 training, 1,015 validation and 1,016 test pairs, and term-level annotations. The Cochrane corpus has 4,459 abstract and plain-language-summary pairs at the paragraph level.
We build three test versions of each:
- clean;
- randomly perturbed, sampled from $S(x)$;
- adversarially perturbed by a stronger, different attack. This attack runs a BBAEG-style staged greedy search that decodes every candidate and scores the true output fact loss. It is affordable only at test time.
Using a different attack at test time guards against a model that has merely learned to resist its own training attack.
The primary metric is critical-fact consistency. We extract numbers with units, UMLS-linked entities and negation cues from the output for the clean input, and report the fraction that survive unchanged in the output for the perturbed input. A dose change counts as a failure even if the output is fluent. We also compute the same fact recall against the reference. Secondary metrics are:
- Flesch-Kincaid grade;
- 4-gram overlap with the source, which detects copying of jargon;
- ROUGE and BARTScore, as used in BioLaySumm 2023;
- MedLane's AScore, HIT and CWR.
Cost is reported as GPU-hours to convergence and as passes per attacked example. Two clinicians will blindly rate 200 perturbed-input outputs per system for clinically consequential errors.
Baselines
The naive reference is BART-base fine-tuned with ordinary cross-entropy on clean pairs. This is the setting where Devaraj et al. report Flesch-Kincaid 13.44 and source 4-gram overlap 0.79 on Cochrane. We expect its fact consistency to drop sharply under perturbation. No reported result was found on Lacuna for any simplifier's robustness to perturbed inputs on MedLane or Cochrane, so we measure this reference ourselves.
The strongest existing clean-quality systems are two. DECLARE (Luo et al.) is a modular pipeline: a PubMedBERT complex-word locator, a dictionary-based interpreter, and a restricted polisher. It reports a 7.6% AScore improvement over BERT-MT on MedLane. No absolute AScore was found on Lacuna, so we will rerun it. The second is the unlikelihood model UL-Cochrane, with reported Flesch-Kincaid 11.97 and source 4-gram overlap 0.49. Both are relevant for two reasons. They define the quality we must not lose, and DECLARE's dictionary-bound locator is exactly the kind of component BBAEG suggests is brittle to typos and number rewrites. We will run both on the same splits and on all three perturbed test versions. Our method beats them if it matches their clean quality within 1 ROUGE point and 0.5 grade level while scoring clearly higher in fact consistency on the adversarial test set.
The strongest robustness competitor is random augmentation. It trains on samples drawn uniformly from the same set $S(x)$, with the same number of augmented examples and the same total GPU budget.
We add an inference-time defense: Dynamic Attention. It attenuates the most-attended tokens with a random mask size and reports up to 33% better robustness than defensive dropout on BERT, GPT-2 and T5. It needs no retraining. We apply it to the naive model.
As a cost ceiling, we run full adversarial training with the decode-every-candidate attack on a 20% subset. This shows how much of the expensive method's gain the cheap version recovers.
Two ablations isolate the key ingredients. The first sets $\lambda = 1$, so perturbations are chosen by damage to the whole output rather than to critical facts. The second picks perturbation sites at random but keeps the rest of the pipeline. All training runs share the same learning-rate search, three seeds and a matched wall-clock budget.
Why not just…?
Why not just augment with random clinical noise? This is the real threat. Random samples from $S(x)$ may cover most realistic variation, and they cost nothing to generate. We think targeting matters because rare, high-stakes substitutions, such as a brand name or a written-out dose, are few in any random sample. BBAEG's result, where only 8% of words were perturbed, suggests that small, well-chosen edits dominate. That evidence is from classification, though. The matched-budget comparison settles the question. If random augmentation comes within noise of our method on adversarial fact consistency, we will report that and recommend it, and the contribution shrinks to the benchmark and the metric.
Why not just normalize the input first, expanding abbreviations, fixing typos and converting numbers? Normalization tools are themselves dictionary-driven. Luo et al. document why dictionaries fail on ambiguous abbreviations such as "TLC", which can mean total lung capacity or thin layer chromatography. A normalizer error passes silently into the simplifier. We include a normalization-then-simplify pipeline as a baseline. The perturbed test versions show whether its errors add up or cancel out.
Why not just prompt a frontier LLM, which is presumably robust to rephrasing? Giammanco et al. find that chatbots often miss the 6th-grade target. BioLaySumm 2023 found prompted LLMs less factual than a plain BART-base baseline. Yang et al. show that layperson accuracy at judging condition severity is only 38.4% on the original radiology text, rising to 52.3% with their self-correcting agent workflow. Plain rewording is therefore not enough, and readers are overconfident. We will still report a few-shot LLM on the perturbed MedLane subset as a reference, with its fact consistency measured the same way.
Decisive Checks
The idea stands or falls on three results on the adversarial MedLane and Cochrane test versions.
- Fact-targeted cheap adversarial training must beat random augmentation at matched GPU-hours on critical-fact consistency. The $\lambda = 1$ ablation must lose part of that gain, which shows the fact weighting matters.
- The cheap attack must recover most of the decode-based ceiling's gain at a small fraction of its cost.
- Clean readability and ROUGE must stay within the bounds set by DECLARE and UL-Cochrane.
If (1) fails, the targeted mechanism is unnecessary. If (3) fails, robustness is being bought with lower-quality simplification, and the method should not be used for patient-facing text.
References
- BBAEG: Towards BERT-based Biomedical Adversarial Example Generation for Text Classification
- Towards Improving Adversarial Training of NLP Models
- An LLM-Enhanced Adversarial Editing System for Lexical Simplification
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Paragraph-level Simplification of Medical Texts
- Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
- BioLaySumm 2023 Shared Task: Lay Summarisation of Biomedical Research Articles
- Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
- Evaluating the Quality and Readability of Generative Artificial Intelligence (AI) Chatbot Responses in the Management of Achilles Tendon Rupture
- Two-Pronged Human Evaluation of ChatGPT Self-Correction in Radiology Report Simplification