Refractory Emission Layers for CTC Speech Recognition

Connectionist Temporal Classification (CTC) is the fast, non-autoregressive workhorse of speech recognition. At every audio frame it independently decides between emitting a character or phone and emitting a "blank" meaning nothing new here. Because of that independence, CTC has no notion of time since its last emission. It cannot express that a new phone rarely begins 20 ms after the previous one, or that after a long pause a new token becomes likely. Neurons face the same problem and solve it with a refractory period: after firing, the firing threshold jumps up and then decays back to baseline. Firing becomes briefly hard or impossible, and afterwards the neuron behaves memorylessly again. We propose replacing CTC's per-frame blank/non-blank decision with exactly this mechanism. The decision to emit becomes a learned threshold crossing, and the threshold is raised by each emission and then relaxes.

The payoff is a specific kind of duration model. Classical explicit-duration models, such as hidden semi-Markov models (HMMs whose states carry an explicit distribution over how long they last), must enumerate every possible duration up to a hard cap, which is expensive and clips long tails. A refractory hazard is different. It departs from memorylessness only for a few frames after an event and is geometric beyond that. Exact training therefore needs only a short "frames since last emission" counter, and the long tail is never truncated. The claim is that this minimal break of frame independence, which conditions on elapsed time but not on token identity, reduces CTC's insertion and timing errors. It keeps greedy left-to-right decoding and adds no second forward pass or autoregressive decoder.

Core Claim

Let $a_t$ be a scalar "emission drive" produced by the encoder at frame $t$, and let $c_t$ be the number of frames since the last non-blank emission, capped at a small window $D$. The probability of emitting a token at frame $t$ is

$$h_t = \sigma\big(a_t - \theta(c_t)\big), \qquad \theta(c) = \theta_0 + \Delta\,\rho^{\,c}\ \text{for } c < D,\quad \theta(c)=\theta_0\ \text{for } c \ge D,$$

where $\sigma$ is the logistic function, $\theta_0$ is the resting threshold, $\Delta \ge 0$ is the threshold jump after an emission, and $\rho \in (0,1)$ is its decay rate. All three are learned, possibly per layer or per token class. If an emission occurs, a separate softmax chooses which token. The implied gap between emissions has probability $P(d) = h_d \prod_{k<d}(1-h_k)$.

The parameters span a range of behaviours. A large $\Delta$ gives an absolute refractory period, like the Poisson-with-dead-time model of The Poisson process with dead time captures important statistical features of neural activity. A moderate $\Delta$ gives a relative refractory period. Setting $\Delta=0$ recovers ordinary frame-independent emission timing.

Each token is emitted exactly once, at one frame. This is the "spike" view of CTC made structural instead of emergent. Training marginalizes over emission times with an exact forward–backward pass over states (frame $t$, tokens emitted $u$, counter $c$). The cost is $O(TUD)$ with $D$ of a few frames after the usual 4x subsampling, versus $O(TU)$ for CTC.

What is new here is the combination of three things. The first is a spike-triggered threshold, taken from Incorporating the Refractory Period into Spiking Neural Networks through Spike-Triggered Threshold Dynamics (RPLIF), used as the output layer of a sequence-level recognition objective rather than as an encoder neuron. The second is a duration family that is non-memoryless only inside the refractory window, so exact dynamic programming needs no hard maximum duration. The third is that the encoder's drive $a_t$ keeps integrating input during refractoriness, rather than being clamped.

Relation to Prior Work

Three papers supply the motivation. CR-CTC treats CTC's single-frame spikes as a pathology. It smooths them with a consistency loss between two heavily masked views, which needs two encoder passes and provides no model of why emissions should be spaced. Spike Window Decoding treats the same spikes as an asset, decoding only small windows around them for a 1.76–2.17x speedup, but it changes nothing at training time. Our layer takes the second view seriously during training: spikes are fine, but their spacing should be modelled. If it works, it also keeps the sparse spike structure that windowed decoding exploits.

Explicit duration modeling supplies the objective and the main contrast. Segmental Recurrent Neural Networks for End-to-End Speech Recognition scores whole variable-length segments with a segmental conditional random field. It reached 17.3% phone error rate (PER) on TIMIT against 18.4% for CTC in the same paper, but it pays roughly quadratic lattice cost and caps segments at 300 ms. Multitask Learning with CTC and Segmental CRF cheapens the segment representation but keeps CTC and segmental heads with conflicting assumptions side by side. Deep Explicit Duration Switching Models for Time Series and Bayesian Inference in Recurrent Explicit Duration Switching Linear Dynamical Systems show that duration priors prevent spurious state flicker, raising adjusted Rand index (ARI) from 0.10 to 0.53 on dancing-bee data. They do this with abstract counters bounded by explicit minimum and maximum durations. We borrow the counter idea, but let it run only through the refractory window, and we parameterize the hazard as a decaying threshold rather than a categorical table.

Refractory modeling supplies the mechanism and one constraint. RPLIF reports that hard clamping blinds low-latency networks to input, while threshold scaling does not; it reaches 82.40% on CIFAR10-DVS at $T=8$. This is why our encoder drive is never zeroed. SPikE-SSM shows that reset-refractory dynamics can be resolved in parallel over 16,384 steps. It motivates, but does not supply, a parallel scan for our layer. Our $O(TUD)$ dynamic program is already cheap, so we do not depend on that algorithm.

The contrast with all of these is that none places a refractory hazard on the emission decisions of a trained speech recognizer.

The break of frame independence is deliberately partial. Token identity remains conditionally independent given the encoder, and only emission timing depends on history. We state this as a limitation rather than hide it.

Research Program

We will first build the refractory emission layer as a drop-in replacement for the CTC output. It consists of a scalar drive head, a token softmax, and the learned threshold $(\theta_0, \Delta, \rho)$, trained with the exact forward–backward pass above and decoded greedily by a left-to-right scan that tracks $c_t$. We will then analyse what the model learns. On TIMIT, we will compare the learned threshold curves with minimum phone durations and inter-onset intervals from the forced alignments. We will also test whether per-class thresholds separate short stops from long vowels and silence.

Finally, we will stress the model where the independence assumption should hurt most. The first stress setting is unidirectional (streaming) encoders, which lack future context; Advancing Connectionist Temporal Classification With Attention Modeling reports vanilla unidirectional CTC at 29.60% word error rate (WER) on Cortana data, a much larger gap than in the bidirectional setting. The second is additive-noise test conditions, where RPLIF's noise robustness suggests refractory suppression of spurious events should help. The third is low-resource training subsets.

Baselines

The naive reference is vanilla CTC with the identical encoder, data, and schedule. This is the $\Delta=0$ special case, so any gain is attributable to the threshold dynamics.

The strongest existing method on this question is CR-CTC CR-CTC. It attacks the same symptom, CTC peakiness, with the same non-autoregressive decoder. It reports 4.35% WER on LibriSpeech test-other (from 5.72% vanilla CTC) and 4.02% character error rate (CER) on AISHELL-1 (vs. 4.8%). We will reproduce it with the same Zipformer-style encoder and parameter count. We will compare at equal training epochs and also at equal GPU-hours, since CR-CTC runs two encoder passes. Each method gets the same hyperparameter search budget, for example the same number of trials over loss weights and masking ratios for CR-CTC versus over threshold initialisations for ours. Beating it means matching its WER/CER at lower training compute, or improving on it when the two are combined. We expect them to be complementary, because one regularizes representations and the other structures emission timing.

Two further baselines isolate the key ingredient. The first applies a fixed dead time after training: a vanilla CTC model whose greedy decoder suppresses emissions for $N$ frames after each emission, with $N$ tuned on dev data. The second trains with a hard absolute refractory period ($\Delta\to\infty$, no relative phase), which is the clamping-style design RPLIF argues against.

A third baseline tests whether the refractory parameterization matters, or merely any duration prior. It is a categorical explicit-duration emission model in the style of hidden semi-Markov models, with a free distribution over inter-emission gaps up to a tuned $D_{max}$, trained with the same dynamic program.

On TIMIT, the segmental recurrent network is the duration-aware reference: 17.3% PER vs. 18.4% CTC Segmental Recurrent Neural Networks for End-to-End Speech Recognition. We would count approaching it at CTC-like cost as success. Transducers are reported only as an upper reference. No reported result found on Lacuna for a transducer on these benchmarks under our settings, so we will train one ourselves.

Why not just…?

Why not just use a transducer or attention decoder, which already drop conditional independence? They do, and on accuracy alone they may win. We make no claim to beat them. The claim is about the regime where CTC is chosen for greedy, parallel, low-latency decoding. CR-CTC reports that a regularized CTC can approach larger transducers, which makes the CTC regime worth improving. We will report latency alongside accuracy, so a reader can see what the refractory layer buys relative to a transducer at matched decoding cost.

Why not just rely on CR-CTC-style augmentation? This is the real threat. If CR-CTC already removes the insertion and timing errors a refractory hazard targets, our layer will add nothing on top of it. The combined run settles this. If refractory plus CR-CTC does not beat CR-CTC alone, especially on insertion rate under noise and in the streaming setting, we will conclude that augmentation subsumes the idea.

Why not just suppress emissions for a few frames at decode time? It is nearly free, and if it matches the trained layer the proposal collapses to a decoding heuristic. There are two reasons to expect it to fall short. It cannot adapt to input strength, and RPLIF reports roughly 20% higher accuracy than rigid designs under heavy noise. It also applies a constraint the encoder was never trained with. The post-hoc dead-time baseline is therefore the first decisive comparison, and we will stop if it is not beaten.

Why not just let a bidirectional encoder learn spacing implicitly? With full future context, frame-independent outputs may already be well coordinated, so gains could shrink at scale. We therefore predict the largest effect in unidirectional and low-data settings. A null result on large bidirectional LibriSpeech models would be reported as a boundary of the claim, not an anomaly.

Proposed Experiments

The main benchmarks are TIMIT, used for PER, insertion rate, and phone-onset timing error against reference boundaries. LibriSpeech is used for WER on test-clean and test-other, plus versions with added noise at several signal-to-noise ratios. AISHELL-1 is used for CER, including compatibility with windowed decoding: we will report Spike Window Decoding's CER and speedup on top of vanilla CTC and on top of our model, since its paper reports 3.69% from a 4.19% baseline. Each is run with bidirectional and unidirectional encoders, and at full and reduced training data.

Beyond error rates, we report three diagnostics. The first is the distribution of inter-emission gaps compared with forced alignments, measured by KL divergence. The second is training throughput. The third is the learned $(\Delta, \rho)$ per class.

The decisive checks are ordered by how easily they can kill the idea. The trained refractory layer must beat post-hoc dead time and hard clamping on insertion rate. It must beat or match the categorical explicit-duration baseline, which would show that the refractory-then-memoryless family loses nothing to an unconstrained table while avoiding $D_{max}$. It must add a gain on top of CR-CTC in at least the streaming or noisy settings.

As a small generality check on the duration family itself, we will substitute the refractory hazard for the bounded counter in a switching model on the dancing-bee data. The reference is RED-SDS's reported ARI of 0.53 Deep Explicit Duration Switching Models for Time Series. This asks whether dropping the hard maximum duration costs segmentation quality.