Radiative-Return Stress Tests for Rare-Event Language Detectors

Particle physicists have a trick for measuring how something behaves across many collision energies while the machine runs at only one. Sometimes an incoming electron emits a photon just before it collides. That emission lowers the energy available to the collision, and the drop can be computed exactly from the photon's energy. Measuring the photon lets them sort events from a single fixed-energy dataset into a continuous scan. They then divide each measurement by a well-understood reference process recorded under the same conditions, which cancels detector efficiency and luminosity errors (Precise Measurement of the e+e-→π+π-(γ)...). We propose to bring this design, with its measured degradation knob and its ratio to a reference channel, to a familiar problem in NLP evaluation. Detectors for rare linguistic events, such as speech disfluencies (restarts, repetitions and filler words that make up under 6% of tokens), are tuned on one canonical corpus. Their headline scores cannot tell us whether they learned structure or learned lexical shortcuts.

The core idea fits in two sentences. From a single existing test set (Switchboard), we build a continuous family of test conditions by removing a measured amount of word-identity information from the input, and we track how rare-event detection degrades along this scan. Every point on the scan is divided by detection of rule-generated disfluencies placed into the same corpus under the same corruption, so that general input damage cancels. What remains is how much the detector's grip on natural disfluencies depends on lexical cues rather than structure. We then test the question that makes this worth doing: does this ratio curve predict out-of-domain performance better than the in-domain score everyone reports?

Core Claim

Disfluency detection on Switchboard is saturated and possibly overfit. Span classifiers reach 94.1 F1, and 94.2 when syntactic information from dependency parses is fed in through a graph convolutional network, a network that passes information along parse-tree edges (Span Classification with Structured Information for Disfluency Detection in Spoken Utterances). That paper validates only on Switchboard and itself flags that parsers fail on disfluent speech. A 0.1-point gain from syntax cannot tell us whether the syntactic channel does anything useful when lexical cues are weak, which is exactly the situation in other speech genres. Our claim is that a normalized degradation scan exposes this, and that its summary statistic ranks models by their cross-domain F1 on CallHome, SCOTUS and FCIC better than Switchboard F1 does. Those three corpora are the out-of-domain sets used by Adaptive Unsupervised Self-training for Disfluency Detection.

The four input directions carry different weight. The physics work on initial state radiation supplies the method: a scan from one operating point, a ratio to a reference channel, and correction for how the knob's nominal setting maps onto its actual effect. Class imbalance defines the regime, since the events of interest are rare and they thin out further as the scan proceeds, just as hard-photon yield falls at low effective energy. Heterogeneous representation is the object under test, meaning models that combine token embeddings with syntactic channels. Reliance on existing datasets is the problem we address. We deliberately build the stress test from the canonical dataset, because new benchmarks tend to become the next overused benchmark.

Research Program

The knob. We replace a fraction $b$ of word tokens with their part-of-speech tag, which is a coarse grammatical category such as "noun" or "verb". Replacement follows a fixed random order per utterance, so conditions are nested. At $b=0$ the input is the original text. At $b=1$ the model sees only a grammatical scaffold. The scaffold keeps repetition only when the tag sequence repeats, which removes lexical cues such as "uh" and "I mean". Nominal $b$ is not a good measure of information removed, because function words and fillers carry different amounts of signal. We therefore report each condition by its measured effect: the drop in accuracy of a fixed linear probe trained to recover word identity from the detector's own input representation. This mirrors the physicists' unfolding step, which corrects for the gap between the measured photon energy and the true energy shift. We will also give a second parameterization based on removed unigram entropy, and check that the conclusions hold under both.

The reference channel. We take fluent Switchboard utterances and insert synthetic disfluencies whose spans are known exactly. The insertions are repetitions, restarts and filler insertions, generated by rules of the kind used to seed the teacher model in Adaptive Unsupervised Self-training for Disfluency Detection. They play the role of the muon-pair reference process. Any damage the knob does to the model's general ability to read input shows up in both channels. For a detector $m$ we define

$$R_m(b) = \frac{\mathrm{Recall}^{\text{natural}}_m(b)}{\mathrm{Recall}^{\text{synthetic}}_m(b)},$$

where the numerator is recall on human-annotated Switchboard disfluencies at knob setting $b$, and the denominator is recall on inserted synthetic disfluencies at the same setting. We summarize each model by the area under $R_m$ over $b$. A flat curve means natural disfluencies are detected through the same structural route as rule-like ones. A curve that collapses as $b$ rises means the detector relies on lexical cues peculiar to Switchboard. The weak point of the analogy should be stated plainly. Synthetic disfluencies are not natural ones, and that mismatch plays the role of the radiator-function uncertainty. We bound it by varying the generation rules and reporting the spread of $R_m$.

Statistics under thinning. Each bin in $b$ contains the same utterances but fewer detectable events. We therefore report bootstrap confidence intervals per bin and per disfluency type (filler, repetition, repair), and pool adjacent bins wherever the interval becomes wider than the effect being measured.

Models scanned. We scan five detectors: a token-tagging BERT, a span BERT, a span BERT with the parse-graph channel, a parser-free span BERT with an added POS-tag embedding channel, and a self-trained detector. We also include two loss variants: class-weighted cross-entropy, and an expected-F-score loss adapted from The impact of language models and loss functions on repair disfluency detection, which trains directly on a smooth approximation of the F-score. Each variant is trained with three seeds, giving a population of roughly two dozen models for rank correlation.

Baselines

The naive reference point is the current practice: rank models by Switchboard test F1 alone. The strongest reported numbers come from Span Classification with Structured Information for Disfluency Detection in Spoken Utterances: 94.2 F1 for span BERT with the parse-graph channel, 94.1 without it, and 88.2 for token-level BERT. For the self-trained detector, Adaptive Unsupervised Self-training for Disfluency Detection reports 90.3 F1, and 90.5 F1 for supervised BERT trained on 60,000 labeled sentences. We reimplement these models under a shared tokenizer, the same encoder size, and the same hyperparameter search budget per model. The comparison should reproduce the published numbers to within about a point before any scan is interpreted. That paper reports no numeric cross-domain F1 on CallHome, SCOTUS or FCIC (no reported result found on Lacuna for those), so we compute cross-domain F1 ourselves for every model.

The strongest competitor for our predictive claim is a direct out-of-domain proxy: held-out F1 on one of the three cross-domain corpora, used to predict the other two. It uses real shifted data, so it is a fair and demanding bar. Our scan needs no extra corpora. It counts as beating this proxy if it matches or exceeds it in rank correlation. It counts as beating the naive baseline if its Spearman correlation with mean cross-domain F1 is higher than Switchboard F1's, with non-overlapping bootstrap intervals.

The key ablation removes the ratio. It uses the raw natural-disfluency recall curve under the same knob, without the synthetic reference. This isolates whether the reference channel adds predictive power or whether plain corruption curves already suffice. A second ablation drops the probe-based recalibration and scans in nominal $b$.

Why not just…?

Why not just evaluate on CallHome, SCOTUS and FCIC directly? Where those corpora exist, one should, and we do. They are the ground truth for our predictive test. But each shifts genre, speaker population, transcription conventions and disfluency rate all at once. They say that a model degrades, not why. Most rare-event tasks also lack three annotated out-of-domain sets. If the out-of-domain proxy predicts as well as the scan, the scan's value shrinks to diagnosis, and we will report that outcome as it stands.

Why not just ablate by masking words at a few levels? That is our raw-curve ablation. The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution found that combining multiple feature views beat masking by 3% as a training method. It did not test masking as a measurement, so it does not settle this question. The danger is that raw corruption curves mostly measure robustness to garbled input, and a strong model looks good on them for that reason alone. The ratio is designed to remove that shared factor. The ablation will show directly whether it matters.

Why not just train with random POS replacement as augmentation? Augmentation might flatten $R_m$ without improving cross-domain F1. That would be the familiar pattern of fitting the test instead of the task. So we include augmented models in the scanned population. If their areas rise while their cross-domain F1 does not, the scan is gameable, and the claim fails.

Proposed Experiments

First, we reproduce the Switchboard baselines and build the scan, with twelve settings of $b$ and three sets of synthetic-generation rules. Second, the decisive check: we compute the Spearman correlation between each model's area under $R_m$ and its mean F1 across CallHome, SCOTUS and FCIC, and compare it against Switchboard F1, the out-of-domain proxy, and the raw-curve ablation. Third, a diagnostic question about the parse-graph channel. Does its advantage over the parser-free POS channel appear only at high $b$, and does it vanish or reverse on SCOTUS, where parser errors should be worst? The second outcome would confirm the error-propagation concern raised by its authors. Fourth, we compare the expected-F-score loss with weighted cross-entropy by type-level $R_m$ on repairs, the rarest type, to test whether training on the metric preserves structural detection or only moves the decision threshold. As a small generality check, we apply the same scan-plus-reference recipe to the Guardian topic-confusion split, using author-style features inserted into off-topic text as the reference channel. This tests whether the recipe carries over to the 50–60% topic-sponge failure of pretrained models reported there. The project succeeds if the ratio scan ranks models by cross-domain performance better than the canonical score does, and it fails informatively if the out-of-domain proxy or augmentation defeats it.