Can Structured Decoding Stand In for Depth? Rare-Repair Tagging in Shallow Disfluency Models

On-device speech assistants must strip disfluencies from transcripts as the user speaks. Disfluencies are the "I want a- uh, I need a flight" restarts. The models doing this work have to be tiny and fast, and tiny models do worst on exactly the rare, structured events that matter. In Switchboard conversational speech only about 5.9% of words belong to a speech repair (The impact of language models and loss functions on repair disfluency detection). Recognizing a repair means relating a span of words to a later correction, which is the kind of multi-step abstraction that extra network layers usually provide. Each extra layer, though, adds serial latency to every token. This note asks whether a cheap output-side structure can do part of the job that depth would otherwise do. That structure is a conditional random field (CRF): a decoding layer that scores whole label sequences using learned scores for each adjacent label pair, rather than labeling each word independently.

The proposal is to measure this trade directly. We will train families of disfluency taggers with 1 to 12 layers at matched parameter and compute budgets. Each family gets three output heads: a per-word softmax, a CRF whose impossible label transitions are hard-masked, and that same masked CRF trained to maximize the expected F-score on repair words instead of likelihood. The hypothesis is that the structured head's gain is largest for shallow encoders, falls as depth grows, and is concentrated on rare repair labels and long repairs. If the hypothesis holds, latency-bound deployments get a concrete recipe and an "exchange rate" between layers and decoding structure. If it fails, we learn that depth cannot be bought back at the output.

Core Claim

Under a fixed budget on encoder depth, combining three ingredients will recover a substantial share of the rare-repair F-score lost by removing layers: an explicit multi-state repair label set, hard transition masks, and an expected-F training objective computed exactly by the CRF's forward-backward pass. The benefit of these ingredients will shrink as depth increases. Formally, the head-by-depth interaction should be positive and significant on reparandum words, and near zero on frequent, locally detectable events such as fillers.

The claim rests on a specific mechanism in how CRFs fail under imbalance. A CRF trained by maximum likelihood on 94% fluent data learns transition scores that strongly favor staying fluent. A shallow encoder produces weak evidence at each word, so it leans on those transition scores more than a deep encoder does. That reliance pushes rare repair paths out of the best-scoring sequence. Masking illegal transitions removes probability mass that should never exist. Training against expected F, rather than likelihood, stops the fluent-to-fluent prior from dominating the gradient. Together they should let structure help shallow models rather than hurt them on the rare class.

Relation to Prior Work

Two papers motivate the depth axis. Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget fixes compute and parameters (about 20M) and finds a 12-layer narrow model far better than a 3-layer wide one (14.56% vs 19.37% WER). Its evaluations use only linear heads on standard tasks, and we ask whether structured heads change that gap on long-tail labels. Learning conditional random field with hierarchical representations for dialogue act recognition places a CRF over stacked autoencoders and finds that 2 hidden layers work best (83.51%) while 4–5 layers overfit. That result constrains us: depth curves on small labeled corpora can turn over, so we report the full curve rather than assuming more depth is better. That paper also works at the utterance level and ignores class skew, both of which we address.

Three papers supply the decoding and loss machinery. Masked Conditional Random Fields for Sequence Labeling masks illegal transitions with a large negative constant during both training and decoding. It reports that deep BERT encoders produce more illegal paths than LSTMs, up to 40% of errors in some datasets. Its experiments cover full-size encoders on named-entity tasks without extreme skew, and we reuse the masking in a depth sweep on rare disfluency labels. The impact of language models and loss functions on repair disfluency detection shows that an approximate expected-F loss helps a reranker over 25 candidate parses, reaching 0.838 F on Switchboard. We move that objective inside an end-to-end neural CRF, where it can be computed from exact marginals. Loss-sensitive Training of Probabilistic Conditional Random Fields offers related loss-scaled and expected-loss CRF objectives, but tests them only on document ranking. Loss-scaled likelihood serves as our comparison objective. We deliberately do not use the AUC-maximized CRF of AUC-maximized Deep Convolutional Neural Fields for Sequence Labeling. Its cost grows with the cube of the label count, which conflicts with a latency budget, and expected F is the metric the task is actually scored on.

One paper appears to contradict the proposal. Disfluency Detection using a Bidirectional LSTM argues that first-order CRFs cannot capture the long-range link between a reparandum and its repair. It replaces the CRF with a BLSTM, an 8-state repair labeling, and integer-linear-programming cleanup, reaching 85.9 F. We take the objection seriously. We adopt its 8-state labels, which turn part of the long-range structure into adjacent-state constraints, and we stratify all results by reparandum length. If CRF gains vanish for long repairs, the objection is confirmed for that regime. Finally, A comparison between a DNN and a CRF disfluency detection and reconstruction system found a CRF beating a sliding-window network on false starts (36.7% vs 27.5% F). That network was a local MLP, and the paper never varied depth. It is early evidence for our hypothesis, but it does not test it.

Research Program

We will build Transformer encoders over word tokens with $L \in \{1, 2, 3, 6, 12\}$ layers. Width is adjusted so that each family matches about 3.1M parameters, the size of the streaming model in Teaching BERT to Wait. A second series matches training FLOPs instead. All encoders start from the same distillation recipe from a full-size teacher, so depth is the only varying factor in representation quality. A 20M-parameter series tests whether conclusions hold at a larger budget.

Each encoder receives four heads. The first is a softmax with cross-entropy. The second is a linear-chain CRF trained by likelihood. The third is the masked CRF (MCRF) trained by likelihood. The fourth is the MCRF trained by expected F. All heads use the 8-state reparandum/repair label set, with masks forbidding transitions such as entering a repair without a preceding reparandum or interruption.

For a sentence with gold repair-word count $G$, the CRF's forward-backward pass gives the marginal probability $p_t$ that word $t$ is a reparandum word, and $g_t \in \{0,1\}$ is its gold indicator. The training objective is

$$\mathcal{L}_{F} = -\frac{2\sum_t p_t g_t}{\sum_t p_t + G},$$

accumulated over a minibatch rather than per sentence so that sentences without repairs still contribute. This ratio-of-expectations form follows the approximation in The impact of language models and loss functions on repair disfluency detection. It is exact to compute here because linear-chain marginals are exact. A small likelihood term is mixed in for stability, with its weight tuned on the dev set.

The analysis has three parts. First, we estimate the interaction between head and depth on reparandum F, using bootstrap confidence intervals over test conversations. Second, we inspect the learned transition matrices for the fluent-to-fluent bias described above. Third, we perform an "exchange rate" reading: how many layers a masked expected-F CRF head is worth at equal rare-class F, and what that saving means in measured per-token latency on CPU.

Baselines

The naive reference point is a shallow encoder with a per-word softmax trained by plain cross-entropy at each depth. This is the "do nothing" configuration, and it collapses toward the fluent class.

The strongest existing system on this question is the BLSTM with 8-state labels and integer-linear-programming cleanup from Disfluency Detection using a Bidirectional LSTM, which reports 85.9 F on Switchboard. We will reimplement it at each of our parameter budgets, so that its structure and our CRF head compete under identical encoders, data splits, and tuning budgets. For full-scale context we also report its published configuration. The reranker of The impact of language models and loss functions on repair disfluency detection (0.838 F) is quoted as a published reference only, because it depends on parsing and web-scale language models outside our budget. Small-model distillation baselines follow Disfluency Detection with Unlabeled Data and Small BERT Models and Teaching BERT to Wait. The file gives no absolute F for either: Rocholl et al. report only "competitive" F1 alongside 99% size reduction, and Chen et al. report only a 9% relative streaming gain. No reported result was found on Lacuna for absolute Switchboard F of either small model, so we retrain both under our setup.

To isolate the key ingredient, the ablations are the unmasked CRF with likelihood, the masked CRF with likelihood, and the masked CRF with loss-scaled likelihood (following Loss-sensitive Training of Probabilistic Conditional Random Fields). Each is run at every depth with the same search over learning rate and loss mixing weight: 30 trials per cell, three seeds.

Beating the baselines means two things. At one to three layers, the masked expected-F head must exceed the threshold-tuned softmax and the reimplemented BLSTM-plus-ILP on reparandum F, with confidence intervals that exclude zero. That margin must also be larger than the margin at twelve layers.

Why not just…?

Why not just reweight the softmax loss and tune a per-class decision threshold? This is the most serious threat, because rare-class F is very sensitive to the decision threshold, and reweighting fixes the majority collapse described in Identifying comparative sentences in text documents. We therefore give the softmax baseline class weights and a dev-tuned per-label bias, which is the strongest version of this alternative. Thresholding cannot repair inconsistent spans, such as a repair with no reparandum, and the masked CRF error analysis in Masked Conditional Random Fields for Sequence Labeling attributes many errors to such paths. If the tuned softmax matches the CRF head at shallow depth, the claim fails, and we will report that.

Why not just make the small model deeper and narrower, since depth beats width? Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget shows depth wins at fixed parameter count, and our sweep includes that option. The deployment constraint we care about, however, is serial latency per streamed token, which grows with layer count, not parameter count. The CRF head adds a transition table over 8 states, which costs little in measured latency. The experiment reports F against measured latency as well as against parameters. Depth "wins" only if deepening is cheaper in latency than the head at equal F.

Why not just distill from a large teacher or add synthetic disfluencies, and keep the softmax? Distillation and augmentation, as in Boosting Disfluency Detection with Large Language Model as Disfluency Generator, work on the input and representation side, and they are complementary to the head. All our encoders are already distilled. The decisive question is whether the head's gain survives on top of distillation, and we run that comparison directly. Augmentation is left out of the main grid because LLM generators drift toward fluent text and need filtering. We test it once, at the shallowest depth, to check that the gain is additive.

Why not just use integer-linear-programming cleanup on a softmax tagger? It enforces the same legality constraints, but only after training, so the encoder never learns from them. The matched reimplementation answers whether that matters at low depth.

Proposed Experiments

The primary benchmark is Switchboard Treebank-3 with the standard disfluency splits. The metric is word-level reparandum F, reported overall, by reparandum length (1, 2–3, and 4 or more words), and separately for fillers as a control. Illegal-path rate and CPU latency per token are also reported.

The secondary benchmark is the Fisher-derived SSR v2 corpus. There the rare category is false starts, and the published reference points are 36.7% F for a CRF and 52.2% for a two-stage hybrid (A comparison between a DNN and a CRF disfluency detection and reconstruction system). This tests whether a single-stage shallow model with the masked expected-F head can approach a hand-built two-stage pipeline.

A transfer check on Disfluent PhoATIS uses the reported 94.33% PhoBERT disfluency F1 (From Disfluency Detection to Intent Detection and Slot Filling) as a full-size ceiling. It asks whether the depth–structure interaction appears in Vietnamese as well.

The decisive check is the sign and size of the head-by-depth interaction on long reparanda. A positive interaction that is concentrated on repair words and absent on fillers supports the claim that structured decoding substitutes for depth on rare, structured events. A flat interaction means the head is a constant add-on. A gain that disappears on long reparanda confirms the first-order-CRF objection from the BLSTM paper in that regime. A second check concerns the transition-prior mechanism. Under likelihood training, the learned fluent-to-fluent bias should be larger and more harmful at shallow depth. The expected-F objective should remove most of the shallow-depth gap that bias causes, and should matter much less at twelve layers.