Many speech recognizers pair a recurrent neural network with a fixed search graph. The network reads audio frames and outputs, for each frame, a probability over characters or phones, plus a "blank" symbol. The graph turns those frame-by-frame guesses into words. It is a weighted finite-state transducer: a graph whose arcs map input symbols to output symbols and carry costs. Several such graphs, covering the network's blank-and-repeat rules, the pronunciation lexicon and the language model, are merged ahead of time into one search graph by an operation called composition. These systems are fast and accurate on clean audio. They have no model of what happens when the input itself breaks. Over a lossy network or on a stalling edge device, frames go missing, repeat stuck values, or arrive corrupted. The network then gives confident but meaningless posteriors, and the decoder must spend them on some legal word sequence. It has no path that says "this stretch of input is unreliable."
We propose to add that path, and to decide when to take it with a calibrated anomaly detector rather than a fixed cutoff. Concretely, we add "fault arcs" to the small token transducer. A fault arc consumes a frame without advancing the label sequence, or bridges a known gap by letting the language model fill in tokens. The cost of each fault arc is set frame by frame from the tail probability of a reliability score, and that tail is fit online with extreme value theory, which models only the extreme end of a score distribution. The search graph is compiled once. The costs change at runtime, through the same lookup mechanism that already supplies acoustic scores. The claim is that this recovers much of the accuracy lost under input faults while costing essentially nothing on clean audio. We further claim the gain holds even against a network trained on corrupted data, as long as the test faults are of a kind not seen in training.
Core Claim and Positioning
Two papers set up the starting point. EESEN composes a token transducer $T$ for the network trained with connectionist temporal classification (CTC, a training loss that lets the network emit blanks and repeated labels without frame-level alignments) with a lexicon $L$ and grammar $G$, giving $T \circ L \circ G$. It reports 7.34% word error rate (WER) on Wall Street Journal eval92 with a graph of 263MB. Phone Synchronous Decoding with CTC Lattice exploits the fact that 87–88% of frames are blank. It skips them using a fixed confidence rule and gets a 2–3x speedup with no WER loss. Both papers supply the method we build on. Both evaluate only on clean, complete input, and both treat every frame as trustworthy.
The fault-tolerance literature names the specific failure we target. Systems are designed for crash-stop failures, where a component simply halts. They break instead under "gray" failures, where a component hangs or degrades without stopping, as Vernik et al. observed for storage connectors. The Weakest Failure Detector for Eventual Consistency shows that a simple timeout cannot tell a slow channel from a dead one. We borrow only the framing from these papers. A missing frame is a crash-like fault the transport layer can see. A stuck or corrupted frame is a gray fault that only a detector can see. The decoder needs a graded belief about both.
Compilation of WFSTs from Decision Trees constrains the design. It reports that a 7-leaf tree compiled into more than 40,000 arcs, which shows that putting context-rich structure into a graph can make it explode. We therefore add fault structure only to $T$, which has a handful of states per token. Fault costs are never baked into the composed graph.
Mechanism
Each frame $t$ gets a reliability score $s_t$. The score combines the entropy of the network's output distribution with the prediction error of a small recurrent next-frame feature predictor. A large error means the frame does not follow from its recent past, the signature of stalls and corruption.
Above a high threshold $u$, we fit a generalized Pareto tail to scores from recent frames. We follow the extreme-value thresholding of Talagala et al., who fit the upper tail of outlier scores to reach a target false-alarm rate without labeled anomalies. The fault arc's cost is
$$c_t = -\log\Big(1 - \big(1 + \xi\,(s_t - u)_+/\sigma\big)^{-1/\xi}\Big) + \beta,$$
where $\xi$ and $\sigma$ are the fitted tail shape and scale, $(\cdot)_+$ clips negative values to zero, and $\beta$ is a fixed penalty tuned on development data. Ordinary frames therefore make the fault arc expensive, and frames deep in the tail make it cheap.
Adaptive baselines can absorb the very anomalies they should flag, a problem the research file calls threshold inflation. To guard against it, the tail is refit only on frames the decoder did not route through fault arcs. This mirrors Griffin, which updates its threshold only from verified-normal traffic.
Frames the transport layer reports as missing get a different arc. This "gap" arc allows up to $k$ token insertions per missing span, scored by the language model plus a per-token gap cost.
Fault-flagged frames are also excluded from phone-synchronous blank skipping. This prevents the rule from Chen et al. from silently discarding corrupted speech as blank.
Research Program
We build the fault-aware $T_f$ in OpenFst and compose it with standard $L$ and $G$. We then measure the growth in graph size and real-time factor (RTF, decoding time divided by audio duration) relative to plain $T \circ L \circ G$.
We train CTC bidirectional and unidirectional LSTM acoustic models in the EESEN recipe. We then build a fault-injection harness covering five fault types: random frame loss, bursty loss from a two-state Markov loss model, stuck-channel stalls that repeat the last frame, bit-flip corruption of feature vectors, and slowly drifting additive noise. The drifting-noise condition targets the weakness reported in Benchmarking IoT Time-Series AD, where recurrent models degrade under monotonic drift.
The analysis asks three questions. Does the adaptive threshold track drift better than a fixed one? Garg et al.'s finding that thresholding matters more than architecture (An Evaluation of Anomaly Detection and Diagnosis) suggests it should. Does the contamination guard prevent missed faults late in long utterances? And how close does the detector come to an oracle that knows exactly which frames are faulty?
Baselines
The naive reference point is the unmodified EESEN-style $T \circ L \circ G$ decoder run on the faulted input. On clean eval92 the reported figure is 7.34% WER for the character system and 7.87% for the phone system (EESEN). We found no reported result on Lacuna for any decoder under frame loss or corruption on WSJ or Switchboard. Faulted numbers will therefore come from our own runs. Clean numbers must reproduce close to the published ones before any faulted comparison counts.
The strongest competitor combines two standard fixes. The first is multi-condition training: the acoustic model is trained with the same injection harness applied to random loss and noise. The second is feature-level concealment, which interpolates across missing frames. This pair is strongest because it attacks the fault where it enters, inside the network's input, and it needs no change to the decoder. It gets the same training data, the same number of epochs, and the same hyperparameter search budget as our system. Our decoder will also be tested on top of it.
Beating it means two things on eval92. First, a statistically significant WER reduction on held-out fault types, stalls and bit flips, that were never seen in multi-condition training. Second, no loss on the fault types that were seen.
Two ablations isolate the key ingredient. The first gives fault arcs a constant cost tuned on development data, with no detector, which tests whether the extra paths alone help. The second uses a fixed threshold on $s_t$ instead of the online tail fit, which tests the value of adaptive calibration. As an alternative adaptive thresholder we include the fast-plus-slow scheme of BERT-LogAnom, a moving average plus a learned drift multiplier. On HDFS log data that paper reports 90.04% F1 with dynamic thresholding versus 88.41% fixed. That gap is the scale of gain we expect from adaptivity in its own domain, not a target for ours.
An oracle fault mask gives the upper bound.
Why not just…?
Why not just train the network on corrupted audio? This is the real threat. If multi-condition training generalizes across fault types, fault arcs are unnecessary. Our view is that it teaches the network faults it has seen. A stuck channel instead produces input that looks plausible frame by frame, and the network will confidently decode it. The held-out-fault experiment settles the question directly. If multi-condition training alone matches our system on stalls and bit flips, we drop the decoder-side claim and report the detector as a diagnostic only.
Why not just conceal missing frames by interpolation? Interpolation turns a known gap into made-up acoustics that the network then confidently labels. The gap arc instead hands that stretch to the language model, which is the component that actually knows what words are likely. Concealment is part of the strongest baseline, so the comparison is direct. Concealment does nothing for gray faults, whose frames are present but wrong.
Why not just widen the search beam? A wider beam explores more paths through the same graph. It cannot create a path that skips a corrupted frame, and corrupted frames produce peaked, wrong posteriors that a wider beam simply follows. We will report WER against beam width for the baseline to show whether this holds.
Why not detect faults and drop bad utterances upstream? Rejecting an utterance is a crash-stop answer to a partial fault. It throws away the reliable majority of the frames. We report utterance-level rejection as an operating point, with WER computed on the utterances it keeps and the fraction of utterances it discards.
Proposed Experiments and Decisive Checks
The primary benchmark is WSJ eval92, the setting in which EESEN reports its 7.34% figure. Fault rates are swept from 0 to 30% frame loss, alongside stall and corruption densities. Switchboard hub5e00 and the Mandarin 5,000-hour task test whether the effect carries over to conversational speech and to phone-synchronous decoding. That paper reports 10.20% character error rate (CER) for its CTC system on the Mandarin task. We report WER or CER, the rate of decoding failures where no hypothesis survives, RTF, graph size against 263MB, and the false fault-flag rate on clean audio.
Three outcomes decide the proposal. First, clean-audio WER must stay within 0.1 absolute of the baseline; otherwise the method imposes a tax that defeats its purpose. Second, the adaptive-cost system must beat the multi-condition-plus-concealment baseline on held-out fault types. Third, under slowly drifting noise, the adaptive tail with its contamination guard must keep a stable false-flag rate where the fixed threshold drifts upward. If the constant-cost ablation matches the full system, the anomaly thresholding adds nothing, and we will say so.
References
- EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
- Phone Synchronous Decoding with CTC Lattice
- Stocator: a high performance object store connector for spark
- The Weakest Failure Detector for Eventual Consistency
- Compilation of Weighted Finite-State Transducers from Decision Trees
- A Feature‐Based Procedure for Detecting Technical Outliers in Water‐Quality Data From In Situ Sensors
- Griffin: An Ensemble of AutoEncoders for Anomaly Traffic Detection in SDN
- OpenFst: An Open-Source, Weighted Finite-State Transducer Library and its Applications to Speech and Language
- Benchmarking IoT Time-Series AD with Event-Level Augmentations
- An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series
- BERT-LogAnom: Enhancing Log Anomaly Detection with Gated Residual BiLSTM and Dynamic Thresholding