Error Regions, Not Error Tokens: Boundary Distributions for Real-Word Error Detection

A real-word error is a misspelling that happens to be a valid word, such as there for their, or a life for alive. Dictionary lookup cannot catch these, so detectors ask a masked language model whether each word fits its context. They do this one token at a time and output a hard yes/no per token. That design breaks down in exactly the cases that remain unsolved. When two errors sit near each other, each one corrupts the context used to judge the other. When an error spans several words, nobody can say precisely where its edges are, and annotators disagree. On a Chinese fine-grained benchmark, BERT-style taggers fall from about 64 F1 on single-typo sentences to as low as 5.6 on multi-typo sentences (Driving Chinese Spelling Correction from a Fine-Grained Perspective).

Video research faces a structurally similar problem: actions have fuzzy start and end points, and annotators place them inconsistently. That community stopped predicting hard boundaries. Instead it predicts a probability distribution over where each edge lies. We propose to bring that idea to error detection. The detector outputs error regions, each with a distribution over its left and right edges. From those distributions we compute, for every token, a smooth probability that it lies inside an error. That per-token probability is then used in two places. It sets how much each token is masked before correction. It also defines which tokens are masked together when the detector decides whether a region is wrong. The hypothesis is that this repairs the multi-error and multi-word failures without hurting single-error performance.

Core Claim

Pointwise detection has two problems in contaminated regions. It judges each token while its erroneous neighbour is still visible. It also passes the corrector a ragged, independent set of per-token suspicions. Our claim is that treating an error as a region with uncertain extent, and scoring and masking that region jointly, recovers most of the multi-typo collapse. Gains should concentrate on multi-typo and multi-word errors and be near zero on isolated single-token errors. If gains instead appear uniformly, or a simpler structured tagger matches them, the claim is wrong in an informative way. The experiments are designed to tell these outcomes apart.

What Exists and How This Differs

The method borrows from the boundary-modeling side. TriDet detects actions in video. Instead of regressing each boundary to a single point, it predicts a probability distribution over a set of nearby bins and takes the expectation. This reaches 69.3% average mAP on THUMOS14. We reuse its relative-bin head, treating tokens as the time axis.

Several video papers replace hard boundary labels with Gaussian-smoothed targets because annotators disagree. Examples include compressed-video event boundary detection, instance-adaptive intensity modeling for facial expression spotting, and procedure segmentation from automatically collected video. These motivate our training targets.

When the Ground Truth is not True shows that annotator boundary errors are systematic rather than random. It also shows that thresholded F1 penalizes soft predictions. That result constrains our evaluation: we must win on standard exact-match metrics, not only on a tolerant metric we design ourselves.

On the error-detection side, Soft-Masked BERT supplies the correction mechanism. It mixes each token's embedding with the mask embedding in proportion to a detected error probability, $e'_i = p_i e_{mask} + (1-p_i)e_i$. Its ablation shows this beats hard masking. However, its $p_i$ are independent per token and are trained on hard 0/1 targets. We replace them with in-region probabilities derived from boundary distributions.

The ESD/ESC pipeline does reason about spans. It wraps detected errors in hard <s1>…</s1> tags and regenerates only the tagged text, so a single misplaced tag hands the corrector a truncated span. We keep its regenerate-only-the-span efficiency but soften the span edges.

WSpeller shows that typos disrupt Chinese word boundaries and that oracle boundaries add 7.4 F1. Its segmentation is still binary, which is further evidence that boundary information matters.

EGCM detects errors zero-shot by masking each token individually and checking whether the original appears in BERT's top-$k$ predictions. Its authors report that recall is capped by this detection front end. It is the clearest example of the single-token masking that our region-level scoring replaces.

Configurable confusion-set detection flags a word when an alternative is $\mu$ times more probable than it. It reports a 0.105 false-alarm rate on learner text and explicitly excludes multi-word errors such as a life/alive. We generalize its likelihood-ratio test from single words to regions.

Research Program

The detector is a pretrained encoder with a TriDet-style head. At every token position $a$ it predicts three things: a probability $s_a$ that an error region is anchored at $a$, and distributions $L_a(\cdot)$ and $R_a(\cdot)$ over how many tokens the region extends to the left and right. Bins run from 0 up to a small maximum, for example 4. From these we compute, for each token $i$, the probability that it lies inside an error region:

$$q_i = 1 - \prod_a \Big(1 - s_a \Pr_{L_a}[\ell \ge a-i]\,\Pr_{R_a}[r \ge i-a]\Big)$$

Here the first probability applies when $i$ is to the left of the anchor $a$ and the second when $i$ is to the right; both are 1 when $i=a$. The product treats anchors as independent, which avoids double counting overlapping regions. In plain terms, $q_i$ is high if some confident anchor's predicted extent probably covers token $i$.

The training targets for the edge distributions are not one-hot. We smooth each gold edge over neighbouring bins with a width estimated from boundary ambiguity. Where a sentence has several reference annotations, the width comes from their disagreement. Otherwise it comes from the set of equally valid edit alignments between the erroneous and corrected text. Single-token errors with unambiguous alignment therefore keep sharp targets. Only genuinely ambiguous edges are blurred, which follows the instance-adaptive idea from facial expression spotting rather than a fixed global Gaussian.

Detection decisions use joint region masking. For a candidate region proposed by the head, we mask all of its tokens together and compute the likelihood ratio between the best alternative filling and the original text. This removes the contamination that occurs when one error is judged while another is still visible. To keep cost bounded, joint masking runs only on the top few region proposals per sentence.

Correction has two tracks. For equal-length Chinese spelling correction, $q_i$ replaces $p_i$ in the Soft-Masked BERT mixing rule. For English, where real-word errors can change length (a life→alive), we keep the ESD/ESC span-regeneration design with two changes. The hard boundary tags become learned boundary embeddings scaled by $q_i$. The corrector also decodes under two or three edge hypotheses sampled from $L$ and $R$, and keeps the one the language model scores highest.

Training data combines the standard sets with targeted synthetic data. For Chinese, we use Wang271K plus the confusion-table corruption used for Soft-Masked BERT, extended so that adjacent multi-character corruptions appear at a controlled rate. For English, we inject real-word confusions into news text, following the configurable confusion-set paper, including multi-word pairs that the paper excluded.

Baselines

The naive reference point is a fine-tuned BERT token tagger with a tuned hard threshold. It is trained on the same synthetic and gold data and uses the same encoder size, so it isolates what any extra machinery buys.

The strongest pointwise competitor is Soft-Masked BERT, which reports 66.4% correction F1 on SIGHAN against 64.9% for plain BERT. We will retrain it on our exact data mixture so that the only difference is pointwise $p_i$ versus region-derived $q_i$.

The strongest Chinese system in the file is EGCM, which reports 81.6 detection F1 and 79.9 correction F1 on SIGHAN15. We will train it on Wang271K as its authors did and compare under matched encoder size and an equal hyperparameter-search budget of the same number of trials. Beating it means exceeding 79.9 correction F1 on SIGHAN15 overall. More importantly, it means a statistically significant gain on the multi-typo subset, measured with paired bootstrap tests.

ReLM reports 91.2 F1 on ECSpell-LAW and 51.2 zero-shot average F1 on LEMON. It is the out-of-domain reference, and we will report against those numbers on the same splits.

For English, ESD/ESC reports $F_{0.5}$ of 66.9 on BEA-2019. It is the direct hard-span competitor. We rebuild it on our encoder and corrector so the comparison isolates soft versus hard span tags. For real-word detection specifically, the confusion-set system with $\mu=10$ reports a false-alarm rate of 0.105 on CLC-FCE. Beating it means a lower false-alarm rate at equal or better miss rate on the same 18,984 sentences.

Two ablations isolate the key ingredients. The first keeps the full architecture but uses one-hot edge targets; it measures the value of soft boundaries. The second keeps everything but scores regions with independent per-token masks instead of joint masking; it measures the contamination fix.

Why not just…?

Why not just Gaussian-smooth the 0/1 token targets? This is the cheapest transfer from the video literature and a real threat. Smoothing token labels mostly lowers confidence on the neighbours of errors. It does nothing about the detector judging one error while another is visible, which is the mechanism we believe drives the 64-to-5.6 collapse reported by the fine-grained benchmark paper. We run it as a baseline. If it matches the full model on the multi-typo subset, the region machinery is unnecessary and we would report that outcome.

Why not just use a BIO tagger with a CRF and feed its marginals into the soft mask? A BIO tagger labels each token as beginning, inside, or outside a span, and a conditional random field (a model that scores whole label sequences jointly) can produce exactly the kind of in-span probabilities that $q_i$ provides. This is the most serious alternative, and it may capture much of the span-coherence benefit. We include CRF marginals as a drop-in replacement for $q_i$. Two things the CRF lacks remain distinct from it: learned edge widths supervised by annotator disagreement, and joint region masking during detection. If the CRF variant matches us, the honest conclusion is that span structure matters and bin-distribution heads do not. The paper would then be framed around span-level soft masking and joint scoring.

Why not just let a sequence-to-sequence model or LLM rewrite the sentence? Rewriting avoids explicit detection, but the ESD/ESC paper reports that full rewriting spends about 3x more decoding steps on copying correct text. EGCM reports a 6.3x speed gap and more over-correction. We will report a rewriting baseline for accuracy context. Our claim, however, is about detection quality and recall in contaminated regions at span-correction cost, and that is where the comparison is made.

Why not just widen the confusion sets? The fine-grained benchmark shows that contextual errors outside phonetic or visual confusion sets cost ReLM 15 F1 on medical text. Confusion sets also do not address the contamination between adjacent errors.

Proposed Experiments

The decisive analysis stratifies every benchmark by error configuration: single isolated error, two or more errors within a small token window, and multi-token or length-changing errors. The configurations come from the fine-grained Chinese benchmark's multi-typo split, SIGHAN13/14/15, and multi-error sentences in BEA-2019 and FCE.

The prediction is specific. The full model should roughly tie the strongest baseline on isolated errors and win clearly on clustered and multi-token errors. The joint-masking ablation should account for most of the clustered-error gain. The soft-target ablation should account for most of the multi-token edge-accuracy gain.

We report standard exact-match metrics as primary: SIGHAN character- and sentence-level F1, ERRANT $F_{0.5}$, and false-alarm and miss rates on CLC-FCE, NEWS and WIKI-EN. Because hard F1 penalizes soft predictions, we additionally report an edge-tolerant span F1 that gives credit when a predicted edge falls within one token of a reference edge. This metric is reported only as a secondary diagnostic, never as the basis of a claim.

We also measure calibration: whether $q_i$ predicts the empirical rate at which annotators include token $i$ in the error span. A synthetic sweep injects controlled numbers of adjacent real-word errors into clean news text and plots detection recall against error density for all systems. The core claim predicts that the curves diverge as density rises. If they stay parallel, the idea has failed.