Do Co-occurring Errors Blind Real-Word Spellcheckers?

A real-word error is a slip that produces a valid word, such as "form" for "from" or "there" for "their". A dictionary lookup cannot catch it. Modern detectors instead use a masked language model, which is a network trained to guess a hidden word from its neighbors. The detector hides the suspect word, asks the model how likely each member of a small "confusion set" is (for example {their, there, they're}), and flags the word when an alternative is much more probable than what was written. Every published evaluation of this approach tests one error at a time, inside otherwise clean sentences. Real writing is different. Learner essays, clinical notes and speech transcripts often contain several errors close together, and each one corrupts the evidence the detector uses to judge the others.

This note proposes to measure that effect directly and then to fix it. Materials engineers learned that corrosion and fatigue acting together destroy steel far faster than the sum of their separate effects. We borrow their method, a factorial stress test with an explicit interaction term, to ask whether co-occurring errors degrade real-word error detection more than additively. If they do, we expect the damage to be concentrated where several covert (valid-word) errors fall inside each other's context window. We then build a coupled detector that judges all suspect positions in a sentence jointly rather than one at a time. The claim is concrete and falsifiable. Independent per-word testing should fail super-additively when real-word errors cluster, and joint inference over the positions should recover most of that loss. Simpler fixes such as pre-cleaning typos or training on noisy data should not, because a covert neighbor looks perfectly fluent to them.

Why this gap is real

The strongest current system is the fill-mask detector of Transformer-Based Real-Word Spelling Error Feedback with Configurable Confusion Sets. It restricts the masked model's prediction to a configurable confusion set and flags observed word $w$ when some alternative $w'$ satisfies

$$\frac{P(w' \mid \text{context})}{P(w \mid \text{context})} > \mu,$$

where $\mu$ is a user-set threshold. That paper supplies our method and our main baseline. Its own numbers motivate the proposal. On synthetic single errors in clean news text it reports a false-alarm rate of 0.001 and a miss rate of 0.005 at $\mu = 10$. On learner exam scripts (CLC-FCE) the false-alarm rate rises to 0.105, and on Wikipedia revisions it is around 0.082 with about 90% of errors found. A hundredfold jump in false alarms between clean and real text is unexplained. Our hypothesis is that much of it comes from noise in the context, which the method assumes away.

Two further papers suggest that context noise matters, though neither tests detection.

  • Semantics or spelling? Probing contextual word embeddings with orthographic noise shows that a single character swap breaks a common word into subword fragments. Clean-to-noised embedding correlation falls to 0.2–0.5, and 100 words of surrounding context barely help. That study used non-words only. A real-word error stays a single in-vocabulary token, so it shifts meaning without triggering the fragmentation that a model might learn to discount.
  • Assessing the Impact of the Quality of Textual Data on Feature Representation and Machine Learning Models finds that downstream clinical classifiers stay stable below roughly 10% token corruption and then collapse. Its noise is a single scalar rate, not interacting error types. This paper constrains us in two ways. It warns that thresholds can appear abruptly. It also shows that an LLM cleaner (Mixtral 8x7B, 63% detection accuracy) rewrites valid abbreviations, and such false corrections then poison neighboring context.

The materials literature contributes only a lens, not a mechanism. Corrosion-fatigue: a review of damage tolerance models argues that models adding independent damage rates fail because each process changes the conditions for the other. Per-position real-word detection is exactly such a decoupled model. Each position is tested as if every other word were correct.

Core Claim

Let $Q$ be detection quality at a target position. We measure it as miss rate at a fixed false-alarm rate. Let $A$ be the density of non-word noise in the surrounding window and $B$ the density of other real-word errors. We define the interaction

$$I(A,B) = \Delta Q(A,B) - \Delta Q(A,0) - \Delta Q(0,B),$$

where $\Delta Q$ is the degradation relative to a clean context.

We predict three things about independent per-position detection:

  1. $I$ will be positive (super-additive) and dominated by real-word neighbors.
  2. $I$ will grow sharply as covert neighbors fall within a few tokens of the target.
  3. $I$ will be largest for confusion sets that are already ambiguous, such as {country, county}, which the Zesch et al. paper singles out.

We further claim that a joint detector will drive $I$ close to zero at matched compute.

Research Program

The first component is a factorial corruption benchmark built on the clean NEWS and NEWS-BALANCED sets from the Zesch et al. paper. Into each sentence containing a target confusion-set position, we inject two stressors at graded densities and at controlled distances from the target:

  • non-word typos, generated by character edits of the kind Matthews et al. use;
  • additional real-word substitutions drawn from the same confusion-set inventory.

We also include whitespace and apostrophe errors as a third, structurally different stressor. Every cell of the grid is evaluated, which is what lets us estimate $I$ rather than just a single degradation curve. We vary the stressors' ratio and order, because data-driven models in the materials work extrapolate poorly when stressor ratios change. We treat that as a hypothesis to test in text too.

The second component is a mechanism analysis. For each cell we record how the log probability ratio at the target shifts, and how much the target's attention goes to corrupted neighbors. This separates two candidate causes. One is fragmentation dilution, which non-word neighbors should produce. The other is fluent misdirection, which real-word neighbors should produce, because the model treats them as evidence.

The third component is the coupled detector. All confusion-set positions in a sentence are treated as unknowns. We run a small number of rounds of Gibbs-style resampling: each position is redrawn from the masked model's confusion-set distribution given the current values of the others. A word is flagged when its posterior marginal favors an alternative by more than $\mu$. A false alarm at one position is therefore no longer frozen into its neighbors' context. Instead, the positions settle together. The cost is roughly the number of rounds times the number of suspect positions, in forward passes. We report it and match it in the baselines.

Baselines

The naive reference point is the Zesch et al. detector run exactly as published: fixed $\mu = 10$, independent per-position tests, and the same pretrained masked model we use for every other method. On clean NEWS it should reproduce the reported 0.001 false-alarm and 0.005 miss rates. That reproduction calibrates our injection pipeline. The same detector at $\mu = 100$ is a second reference point. The paper reports that it nearly eliminates false alarms on WIKI-EN while missing about 8% of errors.

The strongest competitor is a hardened version of that detector, which a careful practitioner would build first. It combines three changes. A standard dictionary-plus-edit-distance spellchecker removes non-word typos first. The masked model is fine-tuned on sentences carrying our full mixture of injected noise. The threshold $\mu$ is calibrated separately for each confusion set on a noisy validation split. It gets the same training data, the same tuning budget in GPU hours as the coupled detector's development, and the same forward-pass budget at test time. It is the most dangerous rival because it attacks every cause of degradation except one: covert neighbors that look fluent.

Two ablations isolate the key ingredient. The first runs the coupled detector with resampling disabled, so it reduces to per-position tests with identical thresholds and fine-tuning. Any gain then comes only from joint inference. The second gives the per-position detector oracle-cleaned context, with all other errors restored. This bounds how much any context-repair method could gain.

We also run a prompted LLM cleaner as an outside reference, given the clinical evidence that such cleaners over-edit. We report the 63% detection accuracy from Sarwar et al. only as context, since it was measured on a different task.

Beating the hardened baseline means the following. On real learner text (CLC-FCE), the coupled detector must achieve a lower miss rate at false-alarm rates matched to the hardened baseline, by a margin outside bootstrap intervals over sentences. The gain must be concentrated in sentences with multiple nearby errors. A uniform gain would suggest that something other than coupling is at work.

Why not just…?

Why not just run an ordinary spellchecker first, then the real-word detector? For non-word background noise this may genuinely suffice, and we expect it to remove most of the non-word part of $I$. The factorial grid will show this directly in the non-word-only cells. It cannot touch covert neighbors, because they are valid words by definition. Its own wrong corrections also create new real-word errors, much as the clinical LLM cleaner rewrote "mane" and "obs". If the pre-cleaning baseline closes the gap on CLC-FCE as well, then real errors are mostly non-word-driven. We will report that as a negative result for the coupling hypothesis.

Why not just fine-tune the detector on noisy sentences? This is the real threat. Augmentation may teach the model to discount suspicious context. Our counter-hypothesis is specific: a model cannot learn to distrust a neighbor that is fluent and in-vocabulary without effectively solving the neighbor's own detection problem, which is what joint inference does explicitly. The first ablation settles the question. The fine-tuned model is identical in both arms, and only resampling differs. If resampling adds nothing on top of fine-tuning, the coupled detector is unnecessary.

Why not just calibrate $\mu$ per confusion set or per noise level? Calibration shifts the operating point along a fixed trade-off between misses and false alarms. If co-occurring errors make the probability ratio itself less informative, no threshold restores separation. We test this by comparing area under the miss/false-alarm curve, which is threshold-free, alongside fixed-threshold metrics.

Why not just prompt a large language model to fix the whole sentence? It does model the sentence jointly, which is the right instinct. But the only reported evidence in this file shows over-editing of valid domain words. It also gives no per-set configurability, which teachers rely on in the Zesch et al. setting. We include it, report its false-alarm rate on correct words, and do not assume it loses.

Decisive Checks

The cheapest decisive test needs no new model. We stratify the CLC-FCE sentences by how many other annotated errors fall within a few tokens of each target. We then check whether the published detector's false-alarm and miss rates rise with that count, beyond what sentence length and confusion-set identity explain. If they do not, the explanation for the clean-to-real gap lies elsewhere and the project should pivot.

If they do, the synthetic grid on NEWS-BALANCED estimates $I$ with confidence intervals for each stressor pair and distance, and tests whether ambiguous sets fail disproportionately. The final test is transfer to real errors: the coupled detector against the hardened baseline on CLC-FCE, WIKI-EN and WIKI-DE. The German set checks that the effect is not specific to English homophones. Success requires a clear win in the multi-error strata of real text. The synthetic grid alone is not enough.