When a language model is trained with reinforcement learning to simplify text, it receives one score for the whole output sentence. Yet whether a sentence became easier to read often depends on one or two words. The simplification corpus The SAMER Arabic Text Simplification Corpus describes this as a "bottleneck effect": a text is only as accessible as its hardest word. We propose to use a complex word identifier to decide where the training signal goes. A complex word identifier is a model that predicts, for each word in context, how hard it is for a reader. It tells us which source words or multi-word phrases actually need rewriting. For each such span, several sampled rewrites of the same sentence are compared with one another, and the span-level difference is credited only to the tokens that produced that span. Every other token receives the usual sentence-level signal.
The underlying policy-gradient method is REINFORCE Leave-One-Out, or RLOO. It samples $k$ outputs per input and scores each output against the average reward of the other $k-1$. This gives an unbiased, critic-free estimate of how good that output is. We keep that machinery and add a second leave-one-out comparison computed per complex span. The expected payoff is lower gradient variance, meaning less noise in each training update. That in turn should give faster, more stable learning of the specific substitutions that matter. The gain should be largest on long sentences that contain few hard words, which is the typical case in real text.
Core Claim
Sequence-level RLOO applies the same scalar advantage to every token. A token copied verbatim from the source therefore absorbs the reward noise produced by an unrelated word substitution three clauses away. When the reward is dominated by a handful of localized lexical decisions, a leave-one-out baseline computed per complex span should reduce gradient variance roughly in proportion to the number of independent decisions in the sentence. This reduction holds if the reward can be written, approximately, as a sum of per-span terms.
Concretely, sample $k$ rewrites $y_1,\dots,y_k$ of a source sentence $x$. Let a complex word identifier mark spans $j=1,\dots,m$ in $x$ whose predicted complexity exceeds a threshold. Let $S_{i,j}$ be the set of output tokens in $y_i$ that are aligned to source span $j$. Let $r_{i,j}$ be the local reward for that output span, defined as its negative predicted complexity plus a local meaning-fidelity term. Let $R_i$ be the global sentence reward, which covers fluency, meaning preservation, and a soft maximum of word complexity. The advantage for token $t$ of rollout $i$ is
$$A_{i,t} = \Big(R_i - \tfrac{1}{k-1}\sum_{l\neq i} R_l\Big) + \alpha \sum_{j} \mathbb{1}[t\in S_{i,j}]\Big(r_{i,j} - \tfrac{1}{k-1}\sum_{l\neq i} r_{l,j}\Big),$$
where $\alpha$ weights the local term. Tokens outside every complex span receive only the first, unmodified RLOO term.
The variance argument is simple. Treat the span rewards as roughly independent. Then the sequence-level advantage seen by the tokens of span $j$ carries the variance of all $m$ span rewards, whereas the local term carries only span $j$'s own variance. This is where the expected $\sim m$-fold reduction for the local component comes from.
The estimator is not unbiased. Tokens in one span can influence neighbouring spans' rewards through context, and the local term ignores those cross-effects. We treat that bias as something to measure rather than assume away.
Relation to Prior Work
Three papers supply the method, and each stops short of this question.
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs introduces RLOO. It treats each whole completion as a single action and shows it matching or beating PPO with a learned critic on summarization and helpfulness tasks. We reuse its estimator unchanged as the global term. Where it spreads the advantage uniformly over all tokens, we direct part of it to complexity-identified spans.
Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models argues that single tokens are the wrong unit of action for reasoning. Its method, Multi-token Policy Gradient Optimization (MPO), groups fixed blocks of $K$ tokens to stabilize importance ratios. It reports MATH 0.789 against PPO's 0.761. That paper motivates choosing a unit of action larger than a token. Its spans, however, are fixed-width blocks and the reward is still sequence-level. Our spans are linguistically defined by the identifier, including multi-word expressions, and each carries its own baseline.
Sentence Simplification with Deep Reinforcement Learning introduced DRESS. It applied REINFORCE to sentence simplification with a sequence-level SARI, relevance and fluency reward, on LSTM encoder-decoders. It motivates the problem: plain likelihood training copies the input. It also constrains our design, because any reward that pays for simplicity invites the policy to delete content. That is why we keep a meaning term both globally and per span.
The complex word identification literature supplies the reward model and the unit of credit. Complex Word Identification: A Comparative Study between ChatGPT and a Dedicated Model for This Task reports a DeBERTa-large model with Zipf frequency features at Pearson 0.81 on CompLex 2.0. In the same study ChatGPT's ranking collapses to Kendall's tau of −0.079 on randomly sampled pairs, against 0.415 for the dedicated model. This is why the reward comes from a dedicated regressor rather than a prompted LLM.
SemEval-2021 Task 1: Lexical Complexity Prediction shows that multi-word expressions are rated harder than single words. This justifies letting spans cover more than one word. It also warns that CompLex over-samples hard words, a bias we address when setting the span threshold.
A Comparison of Techniques to Automatically Identify Complex Words shows the cost of over-flagging. A "simplify everything" baseline reaches recall 0.996 but induces unnecessary rewrites, compared with an SVM at precision 0.7709 and recall 0.8665. In our setting this becomes a concrete design question: how the identifier's threshold trades missed credit against diluted credit.
Research Program
The reward model is the DeBERTa-plus-Zipf regressor, trained on the CompLex 2.0 training split (7,662 instances). A second, independently trained complexity model serves only for evaluation, so the policy cannot simply exploit its own reward model. The policy is an instruction-tuned pretrained language model. It is first fine-tuned on WikiLarge, then optimized with RLOO at $k=4$, the setting at which the RLOO paper reports parity with PPO.
Span identification runs on the source sentence. Output spans are found by monolingual token alignment between source and output. When a span cannot be aligned, as happens with sentence splits or heavy reordering, its tokens simply fall back to the global advantage. This keeps the method well defined on abstractive rewrites while concentrating its effect on lexical ones.
The global reward combines three parts: a soft maximum of predicted word complexity over the output, which reflects the bottleneck effect; a sentence-similarity score for meaning; and a fluency score. The local reward is the aligned span's negative complexity plus a span-level similarity to the source span.
We will analyze three things beyond end-task quality. The first is measured gradient variance: repeated gradient estimates at frozen checkpoints, reported as a function of sentence length and the number $m$ of flagged spans. The second is bias: we compare the proposed estimator's direction against a high-$k$ sequence-level RLOO gradient, whose noise is small enough to serve as a reference. The third is sensitivity to the identifier: we vary its threshold and swap in a frequency-only identifier.
Baselines
The naive reference points are the fine-tuned WikiLarge model with no reinforcement learning and a copy-the-input system. The copy system shows the floor that simplicity metrics must clear. These run on identical test sets and establish whether RL adds anything at all.
The strongest competitor is sequence-level RLOO with exactly the same global reward. It uses the same $k$, the same number of sampled completions, the same KL penalty sweep, and the same learning-rate search budget. It is the right main opponent because it is the most direct simpler version of our proposal. If it matches us, localization adds nothing.
No reported result was found on Lacuna for RLOO, or any variance-reduced policy gradient, on TurkCorpus, ASSET, or Newsela. We will therefore produce this number ourselves. Beating it means one of two things at equal compute. The first is reaching the same held-out complexity and meaning scores in fewer samples. The second is achieving a better final trade-off between complexity reduction and meaning preservation, together with measurably lower gradient variance.
PPO with a learned token-level critic is the second strong baseline, because a critic can in principle learn localized credit by itself. It gets the same reward, the same total generated tokens, and its own tuning budget matched to ours.
DRESS-style single-sample REINFORCE is the historical reference. Sentence Simplification with Deep Reinforcement Learning reports human evaluation on Newsela, but no numerical score table was found on Lacuna. We reimplement its reward on our backbone.
Three ablations isolate the key ingredient. The first uses local span rewards with no leave-one-out baseline, which tests whether the across-rollout comparison matters or only the dense reward does. The second uses random spans of matched count and length, which tests whether identifier-chosen spans matter. The third uses MPO-style fixed $K$-token blocks with per-block leave-one-out baselines, which tests whether linguistic span boundaries beat fixed-width grouping.
Why not just…?
Why not just run sequence-level RLOO with a complexity reward? This is the real threat. On short sentences with one hard word, $m=1$ and our estimator reduces to RLOO. The claim is falsifiable exactly where it should hold. If variance and sample-efficiency gains do not grow with the number of flagged spans and with sentence length, we concede the point. The experiments stratify every result by these two quantities for that reason.
Why not just let PPO's critic learn where credit belongs? The RLOO paper reports that learned critics lag the policy, add memory, and are sensitive to hyperparameters. It also reports that PPO underperforms RLOO in sequence-level alignment. A critic must learn from scratch the complexity structure that the identifier already provides. We test this directly with a matched-budget PPO run and a per-token comparison of critic advantages against our span advantages.
Why not just skip RL and substitute words with an identifier plus a fill-in-the-blank model? Lexical Simplification with Pretrained Encoders (BERT-LS) generates masked-word candidates and ranks them heuristically. It is limited to one-to-one replacements and needs post-processing for tense and number errors. TMU-HIT at MLSP 2024: How Well Can GPT-4 Tackle Multilingual Lexical Simplification? found that re-ranking stages hurt the raw generator in 9 of 10 languages. We include an identifier-plus-BERT-LS pipeline on TurkCorpus and ASSET, and separately report results on multi-word spans, where we expect the pipeline to fail.
Why not just prompt a frontier LLM? Prompted models generate well, but they rank fine-grained complexity poorly (Kendall's tau −0.079 on random pairs). They also give the practitioner no way to target a particular reader's vocabulary. We report a few-shot GPT-4 simplification baseline for reference. Our claim concerns training efficiency for controllable open models, not beating GPT-4 outright.
Proposed Experiments
Training uses WikiLarge sources. Evaluation uses TurkCorpus, whose rewrites are mostly lexical, making it the regime where alignment is reliable. It also uses ASSET, whose splitting and compression rates are 20.2% and 31.2% according to ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations; ASSET tests robustness when many spans fall back to the global signal. Newsela provides grade-level targets, and LexMTurk, BenchLS and NNSeval provide substitution accuracy on the flagged targets.
Metrics are SARI and FKGL, following DRESS. We add the maximum and mean word complexity of the output under the held-out complexity model, a sentence-similarity score for meaning, and training curves of reward against samples. Human evaluation uses pairwise comparisons rather than absolute ratings. The reason is that Comparative judgments are more consistent than binary classification for labelling word complexity found far higher agreement for comparative judgments (Cohen's kappa 0.6775 against 0.3937), and they were also faster. Raters compare pairs of system outputs for difficulty and for meaning.
Four checks decide the outcome.
The first is the variance check. At frozen checkpoints, the localized estimator must show lower gradient variance than sequence-level RLOO. The ratio should grow with $m$, as the span-independence argument predicts.
The second is the efficiency check. At a fixed sample budget, the localized estimator must reach a given held-out complexity reduction sooner, without losing meaning preservation. Held-out complexity is scored by the evaluation model, not the reward model.
The third is the attribution check. The random-span ablation and the local-reward-without-baseline ablation must both fall short of the full method. Otherwise the gain comes from dense reward or span grouping, not from identifier-localized leave-one-out comparison.
The fourth is the bias check. The cosine between our gradient and the high-$k$ reference must stay high. If it degrades as $\alpha$ grows, we report the value of $\alpha$ beyond which the bias cancels the variance savings.
A negative result on the first check would settle the question against the idea, and we would report it as such.
References
- The SAMER Arabic Text Simplification Corpus
- Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs
- Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
- Sentence Simplification with Deep Reinforcement Learning
- Complex Word Identification: A Comparative Study between ChatGPT and a Dedicated Model for This Task
- SemEval-2021 Task 1: Lexical Complexity Prediction
- A Comparison of Techniques to Automatically Identify Complex Words
- Lexical Simplification with Pretrained Encoders
- TMU-HIT at MLSP 2024: How Well Can GPT-4 Tackle Multilingual Lexical Simplification?
- ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations
- Comparative judgments are more consistent than binary classification for labelling word complexity