Where to Put the Rare Tag: Repair-Anchored Tagging for Disfluency Detection

When people speak, they often restart mid-phrase: "I want a flight to Boston, uh, to Denver." To clean such transcripts, a detector has to find the reparandum, the discarded words ("to Boston"), which make up only about 6% of Switchboard words. The usual way to find spans of words is BIO tagging. Each word is labeled as the Beginning of a span, Inside it, or Outside any span. Under this scheme the rarest label of all, B-, sits on the first discarded word. That is the word farthest from the evidence that anything went wrong. The evidence is the repair that comes after the error, usually an approximate copy of it.

We propose that a large part of the reported gap between word-level tagging and span classification comes from this mismatch, not from tagging as such. On Switchboard, a BERT word tagger scores 88.2 F1 and a BERT span classifier scores 94.1 (Span Classification with Structured Information for Disfluency Detection in Spoken Utterances). Our fix moves the rare boundary label to the last discarded word, right before the point where the speaker breaks off. It also trains the tagger directly for the F-score it is judged on. If that gets a linear-time tagger to within a point or two of span enumeration, practitioners gain a cheap, parser-free detector. If it does not, we will have a clean attribution of where the deficit actually comes from.

Core Claim

Word-level disfluency taggers underperform because of two problems that compound. The first is placement: the tag scheme puts its sparsest decision at the reparandum's left edge, where local "rough copy" cues are weakest. The second is the training objective: a cross-entropy loss over roughly 94% O words makes that sparse decision cheap to get wrong. Neither problem alone explains the gap.

The test is a factorial design over three factors on a shared encoder: tag placement (left-anchored BIO versus repair-anchored), objective (cross-entropy, class-weighted cross-entropy, expected-F), and decoding direction. We predict two things. Class weighting by itself recovers recall but loses precision, as earlier work warned it would. Repair-anchored tags combined with an expected-F objective recover most of the span classifier's advantage. A specific error signature distinguishes the explanations: BIO tagger errors should cluster at reparandum left edges and on long spans, and repair-anchored tagging should shift or shrink that cluster.

How This Relates to Prior Work

Several papers motivate the proposal by showing where the evidence lives. Rough-copy repairs mirror their reparandum, as established by A TAG-based noisy-channel model of speech repairs, and shallow repetition cues alone go a long way (Edit Detection and Parsing for Transcribed Speech). Reparanda are recognized retrospectively: Efficient Disfluency Detection with Transition-based Parsing parses right-to-left so the repair is seen first, reaching 85.1 F. Disfluency Detection using a Bidirectional LSTM relies on backward context and an 8-state scheme that also labels repair words, with Integer Linear Programming post-processing to remove illegal tag sequences, reaching 85.9.

We differ from both. The R2L parser changes the parsing algorithm, and the BiLSTM changes the label set without touching class skew. We keep a plain linear-chain tagger and change only where the boundary label is placed and what loss trains it. We also replace ILP post-processing with a conditional random field (CRF) layer, a learned table of tag-to-tag transition scores that makes illegal sequences impossible at decoding time.

Other work constrains the design. The impact of language models and loss functions on repair disfluency detection showed that an approximate expected F-score loss beats log loss under Switchboard's skew, reaching 0.838. It applied that loss only inside a reranker over 25 candidate analyses. We supply the open step: computing the same loss from CRF word marginals inside an end-to-end tagger. Both that paper and Using machine learning to cope with imbalanced classes in natural speech found that forcing sensitivity to the minority class tends to cost precision. The latter also found that ensembles beat simple downsampling for rare interruption points, though it treated detection as point classification rather than span extraction. These findings are why reweighting appears as a baseline and not as our method. Neural Metaphor Detecting with CNN-LSTM Model reports that weighted softmax required dropping the CRF. An expected-F loss over CRF marginals avoids that trade-off.

The span classifier is the work we must answer head on. Its authors read the 88.2 versus 94.1 result as evidence that the task is inherently span-level. They did not test whether that tagger was handicapped by its scheme or its loss. Their method also costs $O(N \cdot L)$ in sentence length $N$ and maximum span length $L$, and its best variant depends on a dependency parser run over messy speech. A further warning sign: Adaptive Unsupervised Self-training for Disfluency Detection reports a supervised BERT tagger at 90.5. The token-level baseline therefore varies by about two points across papers, and our first step re-establishes it under controlled conditions.

Research Program

All taggers share one pretrained encoder (BERT-base, matching the span paper), with a CRF output layer unless stated otherwise.

The repair-anchored scheme labels the last reparandum word E and the other reparandum words I. An optional R label marks the first repair word, which supplies the copy evidence. Nested and consecutive repairs such as "the + [ th- + the ]", which A sequential repetition model for improved disfluency detection flattened into sequential events, stay separable because each one has its own E. Plain Inside-Outside tagging would merge them into one span. We compare left-to-right and right-to-left CRF decoding. With a bidirectional encoder, direction matters only through how the transition scores factor, so we expect a small effect and treat it as a check rather than a contribution.

The objective is a smoothed word-level F-score computed from CRF marginals. Let $p_i$ be the model's marginal probability that word $i$ is in a reparandum, $y_i \in \{0,1\}$ the gold label, and $G=\sum_i y_i$ the number of gold reparandum words. Then

$$\mathcal{L}_F = 1 - \frac{2\sum_i y_i p_i}{\sum_i p_i + G},$$

computed per minibatch with forward-backward. Following the reranking work, we mix it with a small weight of CRF log-likelihood to keep training stable.

Analysis has three parts. We break boundary errors down into left edge, right edge, and interior. We stratify F1 by reparandum length and by repair type: exact repetition, partial-copy repair, and restart or false start. We also run an intervention in the style of Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network: we replace exact copies with paraphrased repairs and see whether each tagger relies on a lexical-identity shortcut.

Baselines

The naive reference points are an all-O predictor, which scores 0 F1 and shows that accuracy is meaningless at 6% prevalence, and an exact-repetition rule within a 6-word window. The rule shows how much of Switchboard surface copying alone solves. The repetition model built on such features reached 83.2 (A sequential repetition model for improved disfluency detection), so any learned tagger well below that is broken.

The direct baseline for our question is the standard BERT BIO tagger with softmax cross-entropy, reported at 88.2 F1 in the span classification paper and 90.5 in the self-training paper. We re-run it with the same encoder, data split, and tuning budget as every other system. Running it both with and without the CRF separates the effect of transition modeling from the effect of the tag scheme.

The strongest existing method is the BERT span classifier with length embeddings, at 94.1 F1 on Switchboard, or 94.2 with a graph network over dependency parses (Span Classification with Structured Information for Disfluency Detection in Spoken Utterances). It is the system that defines the gap. We reimplement the no-parser variant on our encoder and splits, with the same number of hyperparameter trials, and also quote the published figures. Beating it means matching it within the seed-to-seed standard deviation at lower inference cost. We report wall-clock time and the cost scaling with sentence length. A gap larger than two points after all interventions counts as a loss for our hypothesis.

For context we also report earlier non-pretrained systems: transition-based stack LSTMs at 87.5 (Transition-Based Disfluency Detection using LSTMs) and auto-correlational CNNs at 84.5 test (Disfluency Detection using Auto-Correlational Neural Networks). We do not re-run these.

The ablations that isolate the key ingredient form a full grid: BIO and repair-anchored placement, each crossed with cross-entropy, class-weighted cross-entropy, focal loss, and expected-F. The claim requires an interaction: repair-anchored placement with expected-F must beat both single changes by more than their sum of noise.

Why not just…?

Why not just use span classification? It is the published winner, and a reviewer can fairly say the question is settled. Two facts keep it open. The token baseline it beat sits two points below another paper's BERT tagger, so the reported gap is not a controlled measurement. The span method also pays quadratic-ish enumeration cost and, at its best, needs a parser on disfluent speech. Our matched reimplementation settles the matter. If the span classifier still leads by more than two points, we report that tagging has a real span-consistency ceiling.

Why not just reweight the loss or use focal loss? This is the most plausible threat, because it is one line of code. Earlier work found that pushing sensitivity to the minority class costs precision (The impact of language models and loss functions on repair disfluency detection). Reweighting also cannot move the decision to where the evidence is. The ablation grid tests this directly. If weighted cross-entropy with ordinary BIO matches our full method, the honest conclusion is that imbalance alone explains the gap, and we would publish it as such.

Why not just oversample disfluent sentences or add LLM-generated data? Static resampling lost to ensembles and to dynamic schedules in Using machine learning to cope with imbalanced classes in natural speech and Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance. We include sentence-level oversampling as a baseline. Synthetic augmentation (Boosting Disfluency Detection with Large Language Model as Disfluency Generator; no reported number found on Lacuna for its Switchboard score) changes the data, not the tagger, so it is orthogonal to our method. We test whether it stacks with ours, and whether it worsens reliance on copies under the paraphrase intervention.

Why not just use a stronger encoder? A larger encoder may close part of the gap for every method. We therefore repeat the main comparison with ELECTRA and treat the relevant quantity as the gap between methods at fixed encoder size.

Proposed Experiments

The primary benchmark is Switchboard with the standard splits, scored by word-level reparandum F1 over five seeds, plus exact-match span F1.

For restarts, the hardest and rarest subtype, we use the Spontaneous Speech Reconstruction v2 corpus. There a CRF scored 36.7 F on false starts and a two-stage hybrid scored 52.2 (A comparison between a DNN and a CRF disfluency detection and reconstruction system). That hybrid is the bar for our type-stratified analysis.

Cross-domain transfer, training on Switchboard and testing on SCOTUS, CallHome, and FCIC, tests whether repair-anchored tags generalize to SCOTUS, where repetitions are longer. No reported results were found on Lacuna for these sets, so we establish all numbers ourselves.

The decisive checks are three. First, does the left-edge error cluster in the BIO tagger shrink under repair-anchored tagging? Second, does the full method beat weighted cross-entropy alone, beyond seed noise? Third, does it land within two points of the matched span classifier at lower inference time? Pass all three and the "tagging is the wrong formulation" conclusion is overturned. Fail the second and imbalance is the whole story.