Speech recognizers for languages such as Finnish, Turkish and Swahili get much of their accuracy from a second pass. A large neural language model rescores a short list of candidate transcripts, and the candidates often differ by a single suffix. That rescoring model is expensive. Once it is spread across several machines, the slowest machine (a "straggler") sets the response time. Coded computing is a family of methods borrowed from error-correcting codes: it adds redundant computation so the answer can be rebuilt from whichever machines finish first. It has been applied to matrix products, gradients and image classifiers, but never to speech. The obvious transfer is to approximate coded inference, which rebuilds each candidate's score up to some error. It has a specific weakness here. The scores that matter most, those of two candidates differing by one vowel in a case ending, are separated by margins that an approximation error could easily flip.
We propose a decoder that uses approximate coded scores only when they are provably good enough. It commits to the top candidate as soon as the gap between the best and second-best recovered scores exceeds an estimate of the recovery error. When the gap is too small, it waits for exact results or backup workers. The core claim is testable. In morphologically rich languages, most utterances have large score margins and can be committed early from a few workers. The few with small margins are concentrated on suffix-level confusions. Gating on the margin should therefore give tail latency close to that of approximate coding and word error rate close to that of waiting for every worker. We evaluate this with a measurement protocol that reports accuracy and latency percentiles together, so that neither can be traded away unseen.
What exists and what is missing
Three strands of coded computing supply the method. Speeding Up Distributed Machine Learning Using Codes codes linear operations with maximum-distance-separable codes, which rebuild the exact result from any $k$ of $n$ workers. On Amazon EC2 it reports a 40% average and 60% 95th-percentile runtime reduction. ApproxIFER extends coding to black-box neural networks. It encodes queries with Berrut's rational interpolant over Chebyshev points and decodes the soft outputs, reporting 36% to 58% better degraded-mode accuracy than ParM on CIFAR-10 at about 4–7% higher latency. Learning-Theoretic Foundation for General Coded Computing replaces polynomials with smoothing splines and proves error decay of $O(S^3N^{-3})$, where $S$ is the number of stragglers and $N$ the number of workers. It shows this empirically on ViT-B/16. CoCoI codes convolution layers on an 11-node Raspberry Pi cluster.
All four work on fixed-size vision inputs, and all judge success by average reconstruction quality. None asks whether the approximation preserves the ranking among near-identical candidates, which is the only thing a rescoring decoder needs. We keep their encoders and decoders unchanged. What we add is the decision of when their output can be trusted.
The speech strand supplies the regime. Finnish ASR with Deep Transformer Models needs a three-pass decoder that ends with a Transformer-XL rescoring sub-word units. The authors state that single-GPU limits restricted their model search. Language modeling for automatic Turkish broadcast news transcription shows that the choice of unit matters. Lexical stem-endings reach 34.6% WER, against 37.9% for statistical morphs and 41.4% for whole words. Fine-tuning Whisper Tiny for Swahili ASR attributes its remaining 30.62% WER largely to suffix substitutions and word-boundary errors. This is exactly the small-margin class our gate targets.
The evaluation strand acts as a constraint. Better Late Than Never shows that latency metrics can be gamed by systems that simply wait. The lesson for us is that any straggler scheme must be scored on joint accuracy–latency curves per utterance, not on a single mean. No work in the file measures coded or distributed rescoring for any speech task, so we found no reported result on Lacuna for distributed ASR rescoring under stragglers.
Core mechanism
Each utterance produces an N-best list, a short ranked list of candidate transcripts from the first pass. Candidates are padded to a common length, and their input embeddings are encoded across $n$ workers with the rational (ApproxIFER) or spline (general coded computing) encoder. Each worker returns the language-model log-probability of its coded query. After the first $m$ workers respond, the coordinator decodes estimated scores $\hat s_i$ for every candidate. It also computes an error estimate $\hat\epsilon_m$, calibrated offline on held-out utterances as a high quantile of $|\hat s_i - s_i|$ given $m$ responses. The coordinator commits to the top candidate when
$$\hat s_{(1)} - \hat s_{(2)} > 2\hat\epsilon_m,$$
where $\hat s_{(1)}$ and $\hat s_{(2)}$ are the two largest combined first-pass-plus-rescore estimates. The factor of two covers the worst case in which both scores are off by $\hat\epsilon_m$ in opposite directions. If the condition fails, the coordinator waits for more workers, which shrinks $\hat\epsilon_m$. Beyond a deadline, it falls back to exact scoring of only the top few candidates on replicated workers.
The first go/no-go question is whether a Transformer's log-probability varies smoothly enough over interpolated embeddings for $\hat\epsilon_m$ to be small. This is not guaranteed for discrete text. If recovery error at realistic $m$ is larger than typical score margins, the gate never fires and the approach fails. We measure that first.
Research program
We build a distributed rescoring service for three settings, each with its matching language model. Finnish uses YLE broadcast news with a Morfessor sub-word Transformer trained on the Kielipankki corpus, following the three-pass recipe. Turkish uses the Turkish broadcast news database under both stem-ending and statistical-morph units. Swahili uses KenSpeech with Whisper hypotheses rescored by a sub-word model.
We run the service on two testbeds. One is a Raspberry Pi cluster of the CoCoI type. The other is a cloud cluster with injected shift-exponential delays, as in the Lee and Tandon setups. On each we measure four things: the distribution of score margins, how often recovery error flips the ranking, which morphological categories the flips land on, and the joint accuracy–latency frontier of every scheme.
The comparison across unit types is itself a finding. Sub-word units lengthen sequences and make per-query cost more variable. We expect this to fatten the straggler tail and shift the point at which coding beats replication.
Baselines
The naive reference is synchronous rescoring that waits for every worker. It defines the target WER and the worst tail latency. We also run the undistributed rescorer on one node to separate distribution overhead from straggler effects.
The strongest simple competitor is timeout with first-pass fallback. Any candidate whose score has not arrived by a deadline keeps its first-pass score. This costs no redundancy, so it is the baseline most likely to embarrass us.
The second competitor is replication with backup tasks. Each candidate is sent to $r$ workers and the fastest answer is used, with $r$ chosen so its worker count matches ours.
The strongest existing coded method is ungated ApproxIFER, which uses the decoded scores at a fixed recovery threshold. It uses the same encoder as ours with the certificate removed. Its best reported figure, 36% to 58% degraded-mode accuracy over ParM on CIFAR-10, comes from vision. There is no reported ASR number for it on Lacuna. We also run ungated spline coding from the general coded computing paper, for which the file reports no ViT number.
The key ablation is the certificate with fallback but without coding. The coordinator commits once the partial exact scores that have arrived already certify a winner under first-pass fallback for the missing ones. This isolates whether coding contributes anything beyond margin awareness.
All methods get the same N-best lists, the same language model weights and interpolation weights tuned once on the dev set, the same worker count and delay traces, and the same total compute measured in worker-seconds. We count beating the field as follows: at matched worker-seconds, our 95th- and 99th-percentile latencies are below those of full waiting and replication, and WER stays within 0.2 absolute of full waiting, where both fallback and ungated coding lose more than that.
Why not just…?
Why not just replicate each candidate and take the fastest copy? For N-best rescoring this is a real threat. Candidates are independent, and replication has no approximation error. The evidence against it in the file is worker efficiency: ApproxIFER needs $2K+2E$ workers where replication needs $(2E+1)K$, and Lee et al. derive a $\Theta(\log n)$ advantage under exponential tails. Neither result was obtained on speech workloads, where per-query costs vary with sequence length. Our matched worker-second comparison settles the question directly. If replication matches us, we will report that coding does not pay off for rescoring.
Why not just time out slow workers and keep the first-pass score? This is cheap and degrades gracefully. The argument against it is that rescoring earns its gain on exactly the suffix-level confusions that the first pass gets wrong. The Transformer-XL's 3% relative WER gain and the Turkish stem-ending gains come from better modeling of morphology. We will break down WER by whether the error is a suffix substitution. We will also run the certificate-without-coding ablation, which is fallback made margin-aware. If that ablation matches the full method, we will drop coding and publish the gate alone.
Why not just run a smaller model on one machine? Whisper Tiny still reaches 30.62% WER on Swahili after 100 epochs, where a larger Whisper model reaches 27.62%. The Finnish authors report single-GPU limits on their rescorer. A quantized single-node rescorer is included as a baseline point on the frontier, so readers can see whether distribution is needed at all at each latency budget.
Decisive checks
The first check is the go/no-go measurement. On YLE dev we compare the distribution of recovery error against the distribution of top-two margins. If the median margin is not clearly larger than $\hat\epsilon_m$ at $m$ well below $n$, the gate cannot fire often and we stop.
The second check verifies the certificate: on test, the rate of committed-but-wrong decisions should match the calibration quantile.
The third check tests the morphological claim. Held-back utterances should be enriched for suffix-only differences relative to committed ones in Finnish, Turkish and Swahili, and more strongly under statistical morphs than stem-endings.
The final check is the frontier. WER against 95th- and 99th-percentile latency at matched worker-seconds, across all baselines and delay regimes, determines whether the full method, the gate alone, or plain replication is the right design.
References
- Speeding up distributed machine learning using codes
- ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systems
- Learning-Theoretic Foundation for General Coded Computing: The Straggler Setting
- CoCoI: Distributed Coded Inference System for Straggler Mitigation
- Finnish ASR with Deep Transformer Models
- Language modeling for automatic turkish broadcast news transcription
- Fine-tuning Whisper Tiny for Swahili ASR: Challenges and Recommendations for Low-Resource Speech Recognition
- Better Late Than Never: Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation