Real-word spelling errors are typos that happen to produce another valid word, such as "county" typed for "country" or "their" for "there". A dictionary cannot catch them, so modern detectors mask each word and ask a language model whether some other word fits the context better. Two problems keep these detectors from being trusted. First, their confidence is poorly calibrated, meaning a stated probability of 0.9 does not correspond to being right 90% of the time. Because real errors are rare in normal text, even slightly overconfident models flood users with false alarms. Second, the best context models are expensive, yet almost every word in a sentence is fine and needs no deep analysis. The obvious fix is textbook multi-class calibration over the model's word predictions. That fix breaks down here: the "classes" are a 30,000–50,000-word vocabulary, and most word pairs appear too rarely in calibration data to calibrate one by one.
This note proposes a different target. Instead of calibrating the full distribution over the vocabulary, we calibrate only the few probabilities that drive the detector's actions at each word: keep it, suggest a replacement, or escalate the word to a larger model. The calibration map is shared across all (original word, candidate) pairs and conditioned on cheap pair features, so rare confusions borrow statistical strength from common ones. The escalation rule estimates whether the large model would actually fix the word. It does not merely ask whether the small model is unsure, so hopelessly ambiguous words are not sent upstream for nothing. The claim is that this gives open-vocabulary detection with calibrated false-alarm control, at a fraction of the cost of running the large model on every word.
Core Claim
For word-level covert error detection, calibration only needs to hold with respect to a three-action decision. Because the action set does not grow with the vocabulary, one shared, pair-conditioned calibrator can be estimated from a realistic amount of labeled error data even as the candidate space grows from hand-configured confusion sets to the open vocabulary. Methods that calibrate each class or the full simplex, by contrast, degrade as the candidate space grows.
We also claim a second point. Escalation should be driven by an estimate of what escalating gains, meaning the small model is wrong and the large model is right. An estimate of the small model's error alone is not enough. On real error corpora this should cut escalations substantially compared with calibrated-confidence deferral, at equal accuracy.
The decision rule is simple to state. At each word $w$, the small model proposes its best alternative $w'$. Let $p$ be the calibrated probability that $w$ is an error and $w'$ is its correction. Let $c_{fa}$ and $c_{miss}$ be the user-facing costs of a false alarm and a missed error. Let $\lambda$ be the compute cost of escalating one word, and let $g$ be the calibrated probability that escalation changes the outcome from wrong to right. The detector then chooses
$$ a^* = \arg\min \{\, c_{miss}\, p,\;\; c_{fa}(1-p),\;\; \min(c_{miss}\,p,\ c_{fa}(1-p)) - (c_{miss}+c_{fa})\,g\cdot\tfrac{1}{2}\text{-weighted gain} + \lambda \,\} $$
In words: keep, suggest, or pay $\lambda$ to escalate, and escalate only when the expected loss avoided exceeds $\lambda$. The exact gain term is estimated directly as a calibrated expected loss reduction. Only two scalars, $p$ and $g$, need to be calibrated, whatever the vocabulary size.
Relation to Prior Work
The problem is motivated by Transformer-Based Real-Word Spelling Error Feedback with Configurable Confusion Sets. That paper restricts a masked language model to small confusion sets and flags a word when the probability ratio of the alternative to the original exceeds a fixed factor $\mu$. It works on synthetic news but produces false-alarm rates of 0.082 on real Wikipedia errors and 0.105 on learner essays. It also behaves very differently across confusion sets, which is a symptom of uncalibrated scores. We keep its fill-mask scoring but replace the fixed ratio with calibrated, cost-aware actions, and we drop the restriction to configured sets.
Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction shows that calibration matters for over-correction. It uses a calibrated BERT to filter noisy training data and cuts SIGHAN13 false positives from 37.9% to 6.9%. That work is offline data cleaning. We address calibration at test time and cost at inference.
The calibration methods themselves constrain the design. Beyond temperature scaling fits a full $K\times K$ linear map on log-probabilities with off-diagonal regularization. That is infeasible at vocabulary scale and still data-hungry on truncated candidate lists. Top-label calibration bins each predicted class separately, and the paper itself notes this starves rare classes, which describes most word pairs. We reuse the parametric shape of Beta calibration, logistic in $\ln s$ and $-\ln(1-s)$ with the identity map included. Its coefficients become functions of pair features, which is how sharing across pairs is achieved.
The decision-focused view comes from two theory papers. Predict to Minimize Swap Regret for All Payoff-Bounded Tasks defines calibration decision loss, the payoff a decision-maker could gain by recalibrating. Dimension-Free Decision Calibration for Nonlinear Loss Functions shows that auditing decision calibration can avoid dependence on outcome dimension when decisions are smooth. It also shows a $\Omega(\sqrt{m})$ barrier for hard argmax decisions over $m$-dimensional outcomes. We do not claim their guarantees transfer. We use calibration decision loss as an empirical audit metric. Our design avoids the barrier in a cruder way: the calibrated object is two-dimensional rather than vocabulary-dimensional.
The escalation component builds on cascade work in two groups. Calibrated Deferral Routing for Cost-Efficient Guardrails and Forecasting Model Success at Inference Time apply isotonic regression to a scalar small-model score and defer when the small model is likely wrong. They save 55% and 31% of cost on binary guardrails and sequence-level named-entity recognition respectively. Both route whole inputs and predict only the small model's failure. Learning to Cascade and Rethinking Calibration for Early-Exit Neural Networks argue, on vision benchmarks, that calibrated confidence wastes compute on "futile" inputs no model can solve. They propose training for or predicting failure of the whole system instead. We carry this argument to word-level language decisions, where futility is concrete: some words, like the "arrow"/"error" example, cannot be resolved without paragraph context. Model Cascading supplies the uncalibrated routing heuristics we must beat, maximum probability and distance-to-uniform, which save up to 89% of compute on sentence classification.
Research Program
The system has three parts.
The first is a small masked language model that scores every word in one pass. For each position it records the original word's probability, the best alternative from the open vocabulary, which is restricted only by a cheap edit, phonetic, or orthographic plausibility filter, the log ratio between them, the entropy, and the alternative's rank.
The second is a shared calibrator. It maps these scores, plus pair features, to $p$. The pair features are edit distance, homophony, part-of-speech agreement, log corpus frequency of each word, and a pair-frequency embedding with shrinkage toward zero for unseen pairs. The calibrator is a beta-calibration-style logistic model whose three coefficients are linear in the pair features. A small monotone correction on the log ratio guards against misspecification. Calibration labels come from held-out real error corpora, with correct words included at their natural base rate so that probabilities reflect deployment sparsity.
The third is a gain estimator, trained on the same held-out data. It predicts $g$ from the small-model features together with context-difficulty features, following the futility-prediction recipe of Kubaty et al. Its labels are obtained by running the large model offline on calibration data. A word escalates only if the decision rule above selects it. When it does, the large model rescores just that word window in its own tokenization, which sidesteps the shared-vocabulary requirement of token-level routing.
We then analyze how calibration error, false-alarm rate, and calibration decision loss change as the candidate space widens. We compare configured sets, the top-50 candidates, and the open filtered vocabulary, and we split results into buckets by pair frequency.
Baselines
The naive reference points are two single models. One is the small model with the fixed-ratio rule at $\mu=10$ and $\mu=100$, exactly as in the configurable-confusion-set paper. The other is the large model run on every word with the same rule. On WIKI-EN that paper reports a 0.082 false-alarm rate at about 90% detection. On CLC-FCE it reports 0.105 false alarms. On NEWS, raising $\mu$ to 100 nearly removes false alarms but misses about 8% of errors. Beating the naive reference means a lower false-alarm rate at matched detection on WIKI-EN and CLC-FCE, using less compute than the always-large model.
The strongest calibration competitor for the flag/keep decision is class-conditional calibration: top-label histogram binning per candidate pair, run with the official df-posthoc-calibration code. It is strongest because, for frequent confusion pairs, per-pair calibration is essentially the ideal. Alongside it we run Dirichlet calibration with off-diagonal regularization on the renormalized top-$k$ candidates. It is the strongest native multi-class method, and it probes whether full-simplex calibration helps at all once the space is truncated.
The strongest routing competitor is UCCI-style deferral: isotonic regression mapping the small model's margin to its probability of error, with a threshold chosen on validation data. The UCCI paper reports 0.91 micro-F1 at 31% lower cost on its named-entity workload. No reported result was found on Lacuna for any cascade on real-word error benchmarks. We therefore report our own re-implementation. We also include distance-to-uniform routing from Model Cascading.
For fairness, all methods use the same two models, the same calibration/validation/test splits of each corpus, and the same pair-plausibility filter. Every method gets the same hyperparameter search budget, with thresholds and $\mu$ tuned on the same validation split for the same target false-alarm rate. Compute is measured as total forward-pass floating-point operations relative to the always-large model.
Two ablations isolate the proposal's ingredients. In the first, the shared calibrator is stripped of pair features, which reduces it to scalar calibration of the log ratio and isolates the value of borrowing strength across pairs. In the second, $g$ is replaced by the small-model error probability, which isolates futility-aware escalation.
Winning has two parts. The first is a lower false-alarm rate at 90% detection and lower calibration decision loss than per-pair binning on pairs seen fewer than roughly 20 times, while not losing to it on frequent pairs. The second is fewer escalations than UCCI-style deferral at equal final F1.
Why not just…?
Why not just tune $\mu$ separately for each confusion set? This is per-class calibration in disguise and a real threat. For frequent, configured pairs it may match our calibrator. Its weakness is the open-vocabulary regime: the configurable-confusion-set paper already shows false-alarm behavior varying by set, and most open-vocabulary pairs will have almost no calibration examples. The frequency-bucketed comparison settles this directly. If per-pair tuning matches us even on rare pairs, the shared calibrator adds nothing and the contribution reduces to the escalation component.
Why not just apply temperature scaling or isotonic regression to the small model's score? A single temperature cannot correct pair-specific bias, which the Dirichlet calibration paper shows on classwise error, and a single isotonic map on the log ratio is exactly our first ablation. If that ablation matches the full calibrator, the honest conclusion is that pair structure does not matter and the simpler method should be used.
Why not just run one well-trained large model, or one small model trained on cleaner data? The Chinese corpus-refinement result shows that a single cleaned BERT can be strong, at 82.3 F1 on SIGHAN13. A cleaner model does not remove the cost of deep analysis on every word, though, and it does not give a tunable false-alarm dial. We will include a small model trained on the refined corpus as the small component of our cascade on SIGHAN. That tests whether the gains are complementary or redundant.
Why not just defer whenever the calibrated small model is unsure? This is standard cascade practice. The early-exit and learning-to-cascade papers show it spends compute on inputs no model solves, but only in vision. The second ablation measures how many escalations on WIKI-EN and CLC-FCE end with the large model also wrong. If that fraction is small, futility prediction is unnecessary for this task, and we will say so.
Proposed Experiments
For English, calibration, gain-estimator training, and testing use disjoint document-level splits of WIKI-EN and CLC-FCE. NEWS and NEWS-BALANCED serve only as synthetic sanity checks, since the fixed-ratio method already looks near-perfect there. WIKI-DE tests transfer of the English-fit calibrator's pair features to another language. For Chinese, SIGHAN13, 14, and 15 give character-level real-word errors with a large candidate space, and we report sentence-level F1 and false-positive rate against the 6.9% and 82.3 figures above.
The metrics are word-level false-alarm rate at fixed detection, miss rate, F1, top-label and classwise expected calibration error, calibration decision loss under the stated costs, escalation rate, and relative floating-point cost. Everything is reported by pair-frequency bucket and by candidate-space width.
The decisive checks are these. First, as the candidate space widens, per-pair and Dirichlet calibration error should rise while the shared calibrator's stays flat. Second, at matched F1, gain-based escalation should send measurably fewer words upstream than error-based deferral. Third, the full system should reduce the 0.082 and 0.105 false-alarm rates at about 90% detection while spending a small fraction of always-large compute. If either of the first two checks fails, the corresponding ingredient is dropped from the claim.
References
- Transformer-Based Real-Word Spelling Error Feedback with Configurable Confusion Sets
- Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction
- Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration
- Top-label calibration
- Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers
- Predict to Minimize Swap Regret for All Payoff-Bounded Tasks
- Dimension-Free Decision Calibration for Nonlinear Loss Functions
- Calibrated Deferral Routing for Cost-Efficient Guardrails
- Forecasting Model Success at Inference Time: Calibrated Probabilistic Forecasts for Cost-Optimal LLM Cascades
- Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
- Rethinking Calibration for Early-Exit Neural Networks
- Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems