Automatic scores for text simplification tend to overrate the outputs they are asked to judge most often: aggressive rewrites that drop or swap clinical facts. The literature already shows the scores are unreliable on average. SARI, the standard simplification metric, barely tracks factual errors at all. Readability formulas correlate weakly with human ratings. Large language model (LLM) judges favor their own model family. We argue the failure is worse than average unreliability, and more predictable. When a simplification system is tuned against a learned score, it is steered toward exactly the outputs where that score is most wrong. Offline reinforcement learning (RL) calls this failure overestimation on out-of-distribution actions: RL that learns only from a fixed log, with no new trials. The standard remedy there is conservative value estimation, which trains the value estimate to be a lower bound wherever the data give it no support.
This note proposes moving that remedy from the policy to the evaluator. We would train a simplification quality scorer on a few hundred expert-labeled examples. A penalty pushes its scores down on outputs that lie far from anything experts have labeled, especially outputs that current scorers rate highly. The resulting pessimistic evaluator should hold up when a generator is optimized against it. The same machinery also yields an annotation signal: send to experts the outputs where optimistic and pessimistic scores disagree most. The setting is biomedical plain-language adaptation, where expert labels are scarce and factual errors carry physical risk.
Core Claim
The simplification evaluation gap is largely an optimization-induced overestimation gap. For standard learned evaluators, the difference between what the evaluator says and what experts and lay readers actually find should grow with optimization pressure, meaning how hard a generator is pushed to maximize the score. A scorer trained with a conservative, support-based penalty should keep that difference roughly flat. It should do so without losing the ability to rank genuinely better systems. Because the penalty concentrates where labeled data are missing, the gap between a plain scorer and its conservative counterpart should also be a better guide to which outputs deserve expert annotation than random sampling or judge disagreement. Nothing in the file tests either claim.
Why This Idea, and Not the Obvious One
The obvious combination of these three directions is to train a simplification policy with in-sample offline RL, using methods such as The In-Sample Softmax for Offline Reinforcement Learning or the implicit-regularization family in Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization. Both constrain a learned policy to actions seen in the logged data. We think this adds little for language models, for two reasons.
First, at the level of whole sequences, RLHF in an SFT Way shows that offline alignment already reduces to reward-weighted supervised regression. That is an in-sample method, so the policy side is largely conservative by construction.
Second, the unconstrained piece is the reward itself. Every one of these methods trusts the score it weights by. Sentence Simplification with Deep Reinforcement Learning optimized SARI and BLEU with policy gradients and learned to truncate and delete. The later meta-evaluations explain why. Evaluating Factuality in Text Simplification found SARI correlating near zero or negatively with factual errors. APPLS found overlap and embedding metrics blind to simplification and SARI blind to hallucination and incoherence.
So conservatism belongs on the evaluator. The closest prior evaluator work is Evaluation Under Imperfect Benchmarks and Ratings. It builds SynthSimpliEval, shows an LLM jury reaching ICC(2,1) up to 0.754, and trains learnable metrics on jury labels. Its metrics are fit on the distribution of rated outputs and never tested under optimization pressure. The jury's shared biases, such as self-preference, are exactly the kind of error that averaging judges cannot remove.
Research Program
The evaluator $f_\theta(x,y)$ scores a simplification $y$ of source text $x$. It has two outputs, one for factual fidelity and one for lay accessibility. It is built on a fine-tuned encoder over the pair, with optional LLM-judge features as inputs. It is trained on a small expert set $D_E$ of triples $(x, y, h)$, where $h$ is an expert rating.
Borrowing the form of conservative Q-learning, we minimize
$$\mathcal{L}(\theta)=\mathbb{E}_{(x,y,h)\sim D_E}\big[(f_\theta(x,y)-h)^2\big]+\alpha\Big(\mathbb{E}_{x,\;y\sim\mu_\theta(\cdot\mid x)}[f_\theta(x,y)]-\mathbb{E}_{(x,y)\sim D_E}[f_\theta(x,y)]\Big).$$
Here $\mu_\theta$ is an adversarial proposal distribution: for each source, it keeps the top-scoring candidates under the current $f_\theta$, drawn from a large unlabeled pool of generations. The coefficient $\alpha$ sets the strength of pessimism. The first term fits expert labels. The second term lowers scores on the outputs the evaluator itself would pick if maximized, and raises them back on labeled outputs. This is the scorer-side analogue of penalizing the action that maximizes Q.
The unlabeled pool is where the annotation bottleneck is relieved. It is cheap to generate from GPT-4o-class models, LLaMA-3.1-70B, fine-tuned BART and T5, and APPLS-style perturbations of the expert references. The final score is the minimum over the two outputs, so that simplicity gains cannot buy back factual losses.
Two failure modes known from offline RL constrain the design. The first is that $\alpha$ is sensitive: too much pessimism collapses toward "prefer whatever looks like the references," the evaluator analogue of reverting to behavior cloning. The implicit-regularization paper reports this sensitivity for its sparsity coefficient. We therefore select $\alpha$ only on held-out expert labels, by ranking accuracy. The second is that InAC-style methods depend on an accurate model of what the data contain. That is why we define support through labeled pairs and adversarially selected candidates, not through an estimated density over text.
The annotation component uses the disagreement $f_{\alpha=0}(x,y)-f_\alpha(x,y)$ as an acquisition score. In each round we send the highest-disagreement outputs to a clinician for a factual rating. We also send them to lay readers, who answer an objective comprehension question: identifying condition severity, following Two-Pronged Human Evaluation of ChatGPT Self-Correction in Radiology Report Simplification. The objective question matters because that study found lay readers confident but wrong. Self-reported ease therefore cannot serve as ground truth.
This differs from MedVR, which removes annotation through consensus among successful rollouts but needs a verifiable categorical answer. We keep humans in the loop and spend them where the evaluator is least trustworthy.
Data come from three sources. The first is PLABA biomedical abstracts, using expert adaptations and the track's human judgments if they are obtainable, and a newly collected expert set of a few hundred rated outputs otherwise. The second is Newsela with the deletion and substitution annotations of Devaraj et al. The third is SynthSimpliEval, where quality is known to rise with model size.
Baselines
The naive reference is the metric set the field still reports: SARI, BERTScore and Flesch-Kincaid Grade Level. Evaluating the Evaluators reports that Flesch-Kincaid correlates at Pearson 0.16 with human reading ease, against 0.56 for the best LLM. It also reports that SARI ignores factual error. Beating these is necessary but not interesting.
The strongest existing evaluator is the LLM jury of Evaluation Under Imperfect Benchmarks and Ratings: a panel of judges from different model families, with the same models barred from rating their own outputs, reported at ICC(2,1) up to 0.754 on SynthSimpliEval. We also run that paper's learnable metric trained on jury labels. For factuality specifically, the reference point is the Devaraj et al. classifier, with reported F1 of 57.1 for deletions and 19.8 for substitutions on Newsela. We add QAEval, which APPLS found most reliable for factual swaps and negations, and the entailment-based BERT-Checker filter of Data Augmentation for Radiology Report Simplification.
All learned competitors receive the same expert labels, the same unlabeled pool and the same encoder backbone, with an equal hyperparameter search budget on the same held-out split.
The key ablation is our scorer with $\alpha=0$, which isolates the conservative penalty. A second ablation replaces the adversarial proposal $\mu_\theta$ with uniformly sampled candidates. This tests whether the penalty works because it targets high-scoring off-support outputs or merely because it regularizes. A third replaces the support penalty with ensemble pessimism: the mean minus a multiple of the standard deviation across five independently trained scorers or across jury members.
Beating the jury means two things. On the static benchmarks, we must match its correlation with human ratings within noise. Under the optimization-pressure test below, we must show a significantly smaller overestimation gap.
Why not just…?
Why not just use an ensemble of LLM judges and take a pessimistic aggregate? This is the most serious threat. Mean-minus-spread over a jury is a cheap conservative estimator, and it may capture most of the benefit. The file gives a reason to doubt it is sufficient. The jury paper documents self-enhancement bias across model families, and errors that all judges share produce agreement, not spread. Ensemble pessimism cannot penalize a confident, unanimous mistake. A support penalty can, because it keys on distance from labeled data, not on disagreement. The ensemble-pessimism ablation settles this directly. If it matches our scorer's overestimation curve and annotation efficiency, we will report that the contribution reduces to jury pessimism.
Why not just gate outputs with a dedicated factuality metric such as QAEval? Gating helps where the metric is sensitive, and APPLS shows QAEval is good on swaps and negations. However, the same study shows each metric catches one error type while missing others, since perplexity-based scoring caught omissions and LENS caught incoherence. Devaraj et al.'s substitution F1 of 19.8 shows how weak learned factuality detection can be. A fixed gate is also a fixed target that optimization can exploit. We include QAEval gating as a baseline under the same optimization-pressure test. If its gap stays flat, the proposal is unnecessary for factuality, though it may still matter for accessibility.
Why not just constrain the generator with a KL penalty, DPO, or prompted self-correction, and leave the evaluator alone? Constraining the policy limits how far it moves. It does not make the score trustworthy wherever the policy does move. Prompted self-correction reached only 52.3% lay severity accuracy in the two-pronged study. The PLABA reports (Large Language Models for Biomedical Text Simplification: Promising But Not There Yet; BeeManc at PLABA 2024) describe large generators with high completeness but persistent substitutions and drift. We treat policy-side constraints as complementary. In the optimization test, we cross KL-regularized reward-weighted fine-tuning with each evaluator, which shows whether evaluator conservatism adds anything once the policy is already constrained.
Why not just pay for more annotation, or crowdsource it? Random expert annotation is the honest baseline for the acquisition experiment, and we run it at matched budgets. Crowdsourcing clinical factuality is ruled out by evidence in the file. Non-expert labels were noisy enough in Active Label Cleaning to need dedicated cleaning, and lay readers misjudged severity while feeling confident.
Proposed Experiments
The decisive experiment is an overestimation curve. For each evaluator, we optimize generators against it at increasing pressure, using best-of-N selection with N from 1 to 256 and one round of reward-weighted fine-tuning of a LLaMA-class model on PLABA. At each level we measure two things on held-out sources. The first is the evaluator's score. The second is the true outcome: the clinician-rated factual error rate and lay objective severity accuracy, with 38.4% for original text and 52.3% for self-corrected ChatGPT as reference points from the two-pronged study. The prediction is that the gap between evaluator score and true outcome widens with N for SARI, the jury-trained learnable metric and the $\alpha=0$ scorer, and stays near flat for the conservative scorer. For reference, PLABA 2024 reports GPT-4o at SARI 37.72 and LLaMA-3.1-70B at completeness 0.9481; we report these but do not treat them as targets.
A second check guards against pessimism that is merely blanket caution. On SynthSimpliEval, the conservative scorer must still order systems by model size, compared against the reported human-size Pearson of 0.759. On APPLS-style perturbations it must stay sensitive to each error type.
A third check covers annotation efficiency. At budgets of 50, 100 and 200 expert-labeled outputs, we compare disagreement-based acquisition against random selection and against jury-spread selection. The measure is the overestimation gap at N=64 after retraining.
Two outcomes would falsify the proposal: if ensemble pessimism or QAEval gating matches the conservative scorer on the overestimation curve, or if the conservative scorer loses the model-size ordering. Either result would be informative, because it would tell the field which cheaper remedy is enough.
References
- The In-Sample Softmax for Offline Reinforcement Learning
- Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
- RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
- Sentence Simplification with Deep Reinforcement Learning
- Evaluating Factuality in Text Simplification
- APPLS: Evaluating Evaluation Metrics for Plain Language Summarization
- Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification
- Two-Pronged Human Evaluation of ChatGPT Self-Correction in Radiology Report Simplification
- MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning
- Evaluating the Evaluators: Are readability metrics good measures of readability?
- Data Augmentation for Radiology Report Simplification
- Large Language Models for Biomedical Text Simplification: Promising But Not There Yet
- {MaLei} at the PLABA Track of TREC 2024: RoBERTa for Term Replacement - LLaMA3.1 and GPT-4o for Complete Abstract Adaptation
- Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images