Reading the Whole Document Before Choosing a Simpler Word

Lexical simplification replaces hard words with easier ones ("abbreviate" with "shorten") while keeping the sentence grammatical and its meaning unchanged. The best systems that need no human-simplified training data treat this as fill-in-the-blank: a pretrained masked language model, which predicts a hidden word from its surroundings, proposes replacements for the target word. These systems only ever see one sentence. When that sentence is short or vague, the model has too little evidence to tell which sense of the word is meant. It then proposes wrong-sense synonyms, or even antonyms, because "increase" and "decrease" fit the same slots. The strongest current fix gets around this by pulling in outside text. It retrieves 300 sentences from a web corpus that contain the target word, groups them into senses, and runs a generator over every one of them, which is accurate and expensive.

This note starts from a simpler observation. The missing context usually sits in the document the word came from, a few sentences away, and the pipeline throws it away. Computer vision has worked through a closely related problem: convolutional networks see only a small patch at a time. That work taught two lessons that carry over. The first is that cheap global summaries can be broadcast back to every local position without paying for full pairwise attention. The second is that widening the view naively smears out the fine local detail that precise decisions depend on. We propose a training-free lexical simplifier that takes its extra context from the target word's own document. Document evidence is pooled into a candidate distribution and mixed into the sentence-level prediction through a gate that trusts the document only when the local sentence is uncertain. We test whether this matches external retrieval at a fraction of the cost, and whether the gating actually matters compared with the naive alternative of pasting the whole paragraph into the model.

Core Claim

For unsupervised lexical simplification of text that comes from real documents, most of the accuracy that Unsupervised Lexical Simplification with Context Augmentation gains from 300 externally retrieved sentences can be recovered from the handful of sentences in the same document. Recovering it depends on how the context is injected. Our specific, falsifiable claim has two parts. First, simply concatenating the paragraph into a BERT-LS-style input will improve word-sense choice but hurt inflection and local fit. Second, a local-first design recovers both: an uncertainty-gated mixture of a sentence-level prediction with a pooled document-level prediction. If naive concatenation performs as well as the gated design, the second half of the claim is wrong, and we will report that the receptive-field fix in text is simply "use a longer input."

Research Program

The starting point is Lexical Simplification with Pretrained Encoders (BERT-LS). It feeds the original sentence alongside a copy with the target word masked, so the prediction for the blank is conditioned on the word it replaces. It reports gains of about 12 accuracy points on average over static-embedding systems across LexMTurk, BenchLS and NNSeval. BERT-LS supplies our local path unchanged.

The document path borrows the sampling-and-weighting idea of Wada et al.'s context augmentation, but draws contexts from inside the document instead of the web. Suppose the target word $w$ appears in sentence $S$ of document $D$. We collect three kinds of in-document context: the other sentences in which $w$ occurs, the sentences next to $S$, and the few sentences in $D$ whose embeddings are closest to $S$. Each context is run through the same masked cloze, with $w$ masked where it occurs and inserted as a paired cue where it does not. This yields one candidate distribution per context.

These distributions are pooled into a single document distribution $p_{doc}(c)$, weighted by each context's similarity to $S$. The design follows Broadcasting Convolutional Network for Visual Relational Reasoning, which pools a global summary once and broadcasts it back to every position rather than comparing all pairs. That paper reports roughly 60 times faster reasoning than pairwise relation networks on Sort-of-CLEVR. In our setting, the pooled distribution is computed once per document and word type, then reused for every occurrence of the word.

The two paths are combined as

$$\text{score}(c) = \log p_{loc}(c \mid S) + \lambda(S)\,\log p_{doc}(c \mid D),$$

where $c$ is a candidate substitute and $p_{loc}$ is the BERT-LS sentence-level probability. The gate $\lambda(S)$ is a monotone function of the entropy of $p_{loc}$, set by a single temperature that is tuned on unlabeled data only, by maximizing agreement between held-out occurrences of the same word. The gate encodes the vision lesson from Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining and ConvFormer: Combining CNN and Transformer for Medical Image Segmentation. Both keep an explicit local pathway, pooling indices in one case and depth-wise convolutions in the other, because global context alone blurs fine structure. For us the fine structure is inflection and collocation, which only the target sentence can decide.

The final ranking reuses the BERT-LS reranking features: language-model loss, embedding similarity to $w$, and frequency. Following Unsupervised Lexical Simplification for Non-Native Speakers, which found subtitle frequencies (SubIMDB) a better guide to learner vocabulary than web counts, we also test subtitle frequencies as an alternative. One further document-level constraint is added: repeated occurrences of $w$ in the same document receive the same substitute unless their local distributions disagree strongly. The aim is to avoid the incoherent, word-by-word output that Context-Preserving Text Simplification and Context-Aware Document Simplification identify as a failure of sentence-isolated simplification.

The existing lexical benchmarks in the file are distributed as isolated sentences, so evaluation needs a document-grounded test set. We will build one from Newsela, whose articles are professionally rewritten at several reading levels. We align sentences across levels and keep single-word substitutions where the rest of the sentence is unchanged. We then filter these with the same criteria LexMTurk-style annotation uses, and have annotators add further acceptable substitutes for a subset. The labels are used only for evaluation, so the method itself stays unsupervised.

As a secondary arm, we ask whether an architectural change adds anything beyond input-level pooling. We adapt the row-and-column attention of CCNet: Criss-Cross Attention for Semantic Segmentation to a grid of sentences by token positions, so that one axis carries within-sentence attention and the other carries attention across sentences. CCNet reports that this layout gives full coverage with 11 times less memory and 85% fewer FLOPs than dense non-local attention. We add it to a BERT encoder as an adapter trained only with masked-word prediction on unlabeled Newsela and Wikipedia documents.

Baselines

The naive reference points are keeping the original word, which measures how often the benchmark accepts no change, and a frequency-only substituter that takes the most frequent embedding neighbor, in the style of Simplifying Lexical Simplification: Do We Need Simplified Corpora?. That paper reports 68.2% accuracy on the Horn et al. dataset. Every learned method must clearly beat both.

The direct local competitor is BERT-LS run exactly as published, sentence only. It is the method we extend, so any gain must come from the document path. It uses the same encoder checkpoint, the same candidate count and the same reranker as our system.

The strongest existing method is Wada et al.'s context augmentation, which reports 79.9 Accuracy@1 on English TSAR-2022, against 65.7 for the previous best unsupervised system, MANTIS, and 89.3 when ensembled with GPT-3.5. We rerun it on both TSAR-2022 and our Newsela set with the same T5 and DeBERTa components. We also add a compute-matched variant that uses only as many external sentences as our method uses in-document ones. Beating it means matching its Accuracy@1 within one point on the Newsela set at full budget with at least ten times fewer model forward passes, and exceeding it at equal budget. On TSAR-2022, which has no document context, we expect only parity with BERT-LS and state that openly.

The baseline most likely to embarrass the proposal is BERT-LS with the surrounding paragraph concatenated into its input, up to the encoder's length limit. It is run with the same candidates and reranker. The key ablations remove one ingredient at a time. We fix $\lambda$ as a constant to test whether the gate matters, swap weighted pooling for plain averaging, restrict contexts to other occurrences of $w$ only, and drop the consistency constraint. The criss-cross adapter is compared against a Longformer-style encoder of matched size, trained on the same unlabeled text with the same masked-word objective, in the spirit of the LEDpara model used by Context-Aware Document Simplification.

Why not just…?

Why not just paste the whole paragraph into BERT-LS? This is the real threat, and a standard encoder already attends across a few hundred tokens. Our prediction is that concatenation helps sense choice but raises inflection and fit errors, because the masked slot now competes with many unrelated tokens. The concatenation baseline and the fixed-$\lambda$ ablation settle this directly. If concatenation matches the gated mixture on Accuracy@1 and on the morphology error rate, we drop the gating claim and report in-document context as the finding.

Why not just prompt a large language model with the full document? No reported result for a large language model used alone on these lexical benchmarks was found on Lacuna. The file does show that GPT-3.5 gains most from being combined with structured candidate ranking (79.9 to 89.3 Accuracy@1 on TSAR-2022). It also shows that free generation needed entity guardrails in Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text and produced hallucinations in the low-resource medical setting of Context-Aware Automatic Text Simplification of Health Materials in Low-Resource Domains. We include full-document prompting as a candidate generator feeding our ranker, and also as a standalone system. If it dominates standalone, the contribution narrows to cheap, auditable ranking for settings where such models cannot be run.

Why not just keep external retrieval, which already works? It does work, and at full budget it may stay ahead on out-of-document senses. Its cost is 300 sentences and repeated generator passes per target, which rules out interactive or on-device use. The compute-matched comparison and a hybrid that falls back to external retrieval only when the gate reports high uncertainty will show where each source of context pays off.

Proposed Experiments

The main experiment runs every system on the Newsela-derived document set and reports Accuracy@1, precision and potential. It adds three targeted diagnostics: the rate of antonym substitutions, the rate of inflection mismatches, and consistency across repeated occurrences of the same word. Everything is also run on TSAR-2022, LexMTurk, BenchLS and NNSeval with the document equal to the sentence, to confirm that the method falls back to BERT-LS gracefully. Wall-clock time and forward passes are logged per target word.

The decisive checks are three. First, gains should concentrate where the local sentence is uncertain: we stratify results by sentence length and by the entropy of the local distribution, and if the improvement is uniform, the gate is doing nothing. Second, the gap between gated mixing and concatenation on the inflection-error rate tests the dilution hypothesis borrowed from vision. Third, the curve of accuracy against context budget, in-document versus external, shows whether one's own document is the cheaper source of the same information.

A small human study on Newsela paragraphs, rating grammaticality, meaning preservation and simplicity, checks that consistent substitutions read as more coherent rather than repetitive. The criss-cross adapter result is reported whichever way it falls. A null result against the Longformer-style encoder and against input-level pooling would itself be useful: it would show that in text, the receptive-field bottleneck lies in pipeline design rather than in the architecture.