Simplification as Refactoring: Regression-Tested Rewriting of Biomedical Lay Summaries

Writing a plain-language summary of a biomedical paper means changing how every sentence is phrased without changing what any sentence claims. Current systems do both jobs at once, in one generation pass. The documented result on the eLife lay-summary benchmark is a trade-off: the more readable the output, the more it drifts from the facts. Programmers face the same problem when they refactor, meaning they rewrite code so it is clearer but behaves identically. Their answer is not a smarter rewriter. They write regression tests first, change the code in small steps, and throw away any step that breaks a test. We propose the same discipline for lay summarization. First produce a faithful technical summary. Then pin its scientific claims as a suite of small, mechanically checkable "fact tests." Then simplify step by step, accepting a rewrite only if every test still passes and rolling it back otherwise.

The key move is to shrink what the large language model (LLM) acting as judge has to do. It no longer delivers a holistic verdict such as "is this summary faithful?" Instead it fills specific slots: what organism was studied, which way the effect went, how certain the authors were. A deterministic comparator then checks those slots against the pinned values. If the approach works, it would show that the readability–factuality trade-off on eLife comes partly from doing both jobs in one pass, and that separating them with tests moves the trade-off curve outward rather than just sliding along it.

Core Claim

Our claim is that test-gated iterative simplification with rollback reaches a lower reading grade level at a fixed rate of expert-annotated factual errors than three alternatives on the eLife test set: single-pass prompting, holistic judge-and-revise loops, and best-of-N reranking with a factuality metric, all at matched LLM call budgets. We further claim that the advantage comes specifically from the structured tests and not merely from iterating. The mechanism is that slot-level tests make each failure point to a particular sentence and a particular attribute. That targets the documented weakness of LLM judges, which often detect an error but blame the wrong part of the text.

Where This Sits

Three lines of work motivate the gap.

The BioLaySumm shared tasks (2023 overview, 2024 overview) set up the eLife benchmark and the relevance/readability/factuality metric suite. They report that a plain BART-base baseline won on factuality while the more readable systems drifted. Clinician review of LLM summaries of Cochrane reviews (Evaluating large language models on medical evidence summarization) supplies the error taxonomy we test against. That taxonomy covers "certainty illusions," where hedged findings become absolute; swapped patient/intervention/outcome roles; and fabricated claims. The same study also found that longer inputs increased errors.

Neither line of work closes the loop. Generation on eLife is single-pass, as in few-shot GPT-3.5 (MDC), or an open-loop pipeline, as in the MeSH-then-summarize system of Towards Knowledge-Guided Biomedical Lay Summarization. The MeSH work explicitly leaves faithfulness evaluation open.

Two papers supply the method, and each differs from ours in a specific way. MEIC debugs Verilog hardware code with a generator agent, a scoring agent, and rollback to the last stable state when a simulator check or the scorer flags a regression. We reuse the rollback logic. Our "simulator," however, must be built for text, and building it is the main contribution. Pelican breaks image-grounded claims into predicates and runs code against a table of detected objects, cutting hallucinations by 27% on MMHal-Bench. We borrow the decomposition into predicates. What we check against is different: a pinned record extracted from a faithful draft of the paper, rather than a perception table.

Two results constrain the design. SummExecEdit shows that Claude 3 Opus detects injected factual errors at 0.67 but reaches only 0.49 when it must also explain them correctly, and that 45.4% of failed explanations blame the wrong span. Feeding holistic critiques back into a rewriter is therefore risky. Rethinking Code Refinement finds that machine "improvements" to code make it slower about 30% of the time. Iteration without an acceptance gate should therefore be expected to regress.

The closest prior system is Agent-as-Judge for Factual Summarization of Long Narratives. It checks atomic facts against a character knowledge graph and feeds natural-language feedback into refinement, gaining +14.03 factuality on MovieSum. We differ from it in three ways. It refines to repair factuality, whereas we refine for readability under a factuality invariant. Its feedback is free text, whereas our gate is a deterministic pass/fail over categorical slots. It has no rollback.

We also depart from CoBa. CoBa backtracks during token-by-token decoding using token probabilities and embedding distance. It needs access to model internals and cannot see errors such as a flipped certainty level.

Research Program

The system has three stages.

Stage one: faithful technical draft. We write a summary that stays close to the paper, from the abstract plus the introduction and results sections. We prefer faithful over readable at this stage, following the BioLaySumm observation that less abstractive outputs are more factual.

Stage two: building the test suite. An extractor LLM turns each claim in the draft into a record. The record holds the subject, the relation, and the object, together with four categorical attributes: direction of effect (increases, decreases, no effect), certainty level (established, suggested, speculative), the experimental system (for example human cohort, mouse, cell line, or in silico), and any quantity. Each record is checked once against the source paper's results sections. Records that cannot be supported are dropped rather than pinned, so the suite never locks in an error. The pinned records form the regression suite. Certainty level and experimental system are deliberately first-class slots, because these are exactly the attributes that get "simplified away" when an LLM writes for the public. Examples are "in mice" disappearing, or "may contribute" becoming "causes."

Stage three: gated simplification. A rewriter proposes one localized simplification at a time: replace a term, split a sentence, or add an analogy. Extraction then runs again on the edited summary. Categorical slots are compared exactly. Subject and object are compared by an equivalence judgment restricted to the two spans in question. Writing $T$ for the pinned test suite, $s$ for the current summary, $s'$ for a proposed edit, and $G(\cdot)$ for Flesch-Kincaid grade level, the acceptance rule is

$$ \text{accept } s' \iff \text{pass}_t(s') = 1 \ \ \forall t \in T \quad\text{and}\quad G(s') < G(s), $$

and otherwise the system rolls back to $s$ and passes the named failing test to the rewriter as its feedback.

Handling added background. eLife digests legitimately add background context that is not in the paper. Sentences that match no test are therefore labeled "background." They are checked only for contradiction with the source, not for support. This is a deliberate weaker guarantee, and we measure how often it lets errors through.

Stopping and extractor reliability. The loop stops when grade level stops falling or the call budget runs out. Because the extractor is itself an LLM, we run it twice with different sampling and treat disagreement as a failed test. This choice is conservative: it favors rollback. We report the extractor's own slot accuracy against human labels.

Baselines

The naive reference point is the fine-tuned BART on eLife from IITR, which reports ROUGE-1 42.81% and FKGL 10.7, together with the BART-base baseline that ranked first for factuality in BioLaySumm 2023. The file quotes no absolute BARTScore value for that baseline, so we rerun it ourselves.

The strongest single-pass baseline is few-shot prompting of the same backbone LLM we use for rewriting, following MDC. Its GPT-3.5 variant won BioLaySumm 2023 Task 1 with a cumulative rank of 16. No per-metric numbers for it were found on Lacuna.

We also include a fine-tuned small model trained with low-rank adapters and direct preference optimization, following BDA-UC3M. That paper reports that Qwen3 4B led on the AlignScore and SummaC factuality metrics, but no reported AlignScore or SummaC numbers for it were found on Lacuna. The 2024 overview states only that the best systems reached a 10th–12th grade reading level.

The two strongest competitors on our specific question are loops at matched compute. The first is a holistic judge-and-revise loop in the style of NarrativeFactScore. It uses the same backbone and the same maximum number of calls; atomic facts are checked against the source, and free-text feedback goes to the rewriter. It has no structured slots and no rollback. The second is best-of-N reranking: sample N lay summaries with N set to match our average call count, then pick the most readable candidate whose AlignScore exceeds that of the technical draft.

Every system receives the same inputs (abstract plus introduction and results), the same backbone where one applies, and the same prompt-tuning budget on the 241-article validation split.

Two ablations isolate the key ingredients. The first keeps the loop and rollback but replaces the slot tests with a holistic judge score as the gate. This tests whether structure matters. The second keeps the tests but accepts every edit and only reports failures, with no rollback. This tests whether gating matters.

We count as "beating" a baseline a statistically significant reduction in expert-annotated taxonomy errors per summary at matched FKGL, together with no loss in reading grade level.

Why not just…?

Why not just prompt a frontier model once to "simplify but keep every fact"? The Cochrane study found certainty illusions and harmful content persisting in single-pass ChatGPT summaries (Evaluating large language models on medical evidence summarization). The MDC authors themselves conclude that unverified output is not ready for dissemination. Neither result tells us how the latest models behave on eLife, however. That is why the single-pass baseline uses our own backbone and sits on the same readability–error curve. If it matches us, the method is unnecessary.

Why not just critique-and-revise with an LLM judge? This is the closest competitor. SummExecEdit's finding that 45.4% of failed explanations blame the wrong span suggests that free-text feedback will misdirect edits. The code-refinement evidence suggests that ungated iteration regresses. The holistic-gate ablation settles the question directly. If holistic gating with rollback performs as well as slot tests, then our contribution reduces to MEIC-style rollback, and we would report that.

Why not just sample many summaries and rerank? This is a real threat, because it is simple and parallel. Reranking can only choose among whole candidates, and when simplification pressure is high every candidate may carry at least one dropped hedge. Test-gated editing repairs the individual offending sentence instead. The matched-compute best-of-N baseline is the decisive comparison. If it reaches the same frontier, we abandon the loop in favor of reranking with our slot tests as the filter, which is itself a useful result.

Why not just fine-tune with preference optimization? BDA-UC3M shows that preference tuning helps factuality metrics but still trades off against other scores. It also requires open weights and gives no guarantee on a given output. Our method works with API models and can be layered on top of a fine-tuned model. We test the combination as well.

Proposed Experiments

The main evaluation uses the eLife test split of 142 articles from BioLaySumm. We report the official metrics: ROUGE, BERTScore, FKGL, the Dale-Chall readability score, and BARTScore, along with AlignScore and SummaC. Because automatic factuality metrics are exactly what is in doubt, the decisive measure is a blinded expert annotation. Annotators with biomedical training label every sentence of 60 test articles across all systems, using the Cochrane taxonomy of contradiction, certainty illusion, role or attribute swap, and fabrication, plus our "dropped experimental system" category. From these labels we plot reading grade level against errors per summary for every system. We expect the gains to depend on how aggressive the simplification is, so we trace each system's curve by varying its simplification pressure.

The second experiment validates the judge on its own. Following SummExecEdit's executable-edit protocol, we inject controlled errors into eLife reference lay summaries by deterministic span replacement: hedge removal, direction flips, organism swaps, and changed numbers. We then measure both detection and correct localization of the error. Our slot tests are compared against a holistic LLM judge on the same backbone. The reference point is Claude 3 Opus at 0.67 detection and 0.49 joint score on general-domain SummExecEdit. We make no claim of direct comparability, because the domain differs. The hypothesis is that slot tests narrow the gap between detection and correct localization.

The third experiment measures regressions per iteration. For every accepted edit, annotators judge whether it introduced a new error. We compare the gated loop with the no-rollback ablation. By analogy with the roughly 30% machine-refinement regression rate reported for code, we expect a comparable failure rate without gating and a sharply lower one with it.

To guard against the backbone having memorized its test articles, we repeat the main comparison on eLife articles published after the backbone's knowledge cutoff, mirroring the 71-article held-out set of the MeSH paper. We also report the background-sentence contradiction miss rate as a known limit of the method.

The project fails cleanly on either of two outcomes. If best-of-N or holistic gating matches slot-gated refinement at matched compute, structured tests are not earning their cost. If extractor slot accuracy falls below the error rate the tests are meant to catch, the "simulator" is too noisy to serve as ground truth.