When Hesitation Is Situational: Auditing Spoken-Language Evaluators for Disfluency-Type Bias

Language models are increasingly used to score what people say: a job candidate's interview answers, or a patient's responses to an automated intake agent. Spontaneous speech is full of disfluencies, the "um"s, repeated words, abandoned phrases and mid-sentence corrections that speakers produce while planning. Psycholinguistics shows that these are not a stable personal trait. They change with the speaker's situation. Drivers under heavy mental workload abandon more phrases but use fewer filled pauses and repetitions. Interviewees produce twice as many filled pauses when talking to a computer agent as when talking to a human. Speech also shifts under physical load. If a scoring model penalizes disfluency, it may be penalizing the conditions someone spoke under rather than what they said. Those conditions are unevenly distributed: speaking a second language, being interviewed by a bot, or being tired from shift work are not demographically neutral.

The main claim is that the harm comes from the type of disfluency, not its amount. Repetitions are easy for both models and cleanup tools to see through. Revisions and abandoned phrases are not. Revisions are corrections where the discarded words can mislead, and abandoned phrases are false starts. High-load speech shifts the mix toward exactly these hard types. Therefore both the raw "feed the transcript to an LLM" pipeline and the obvious "clean the transcript first" fix should produce score gaps that follow situational load, even when the number of disfluencies is held fixed. We propose a counterfactual audit that tests this directly. It transplants real disfluency structures, with type mixes calibrated to measured load conditions, into fixed answer content. We complement it with a natural experiment on clinical interviews in which the interviewer type varies.

What the literature establishes, and what it leaves open

Three strands motivate the claim. The first comes from psycholinguistics and speech science. Lindström et al. found that under high cognitive workload drivers increase deletions (false starts) while filled pauses and repetitions decrease. This is direct evidence that load changes the composition of disfluency, not just its rate. Kawaguchi et al.'s CIAIR corpus ties driving maneuvers to more disfluency. Weston et al. and Godin et al. show acoustic changes under even light physical load, with large individual variation. Gratch et al.'s DAIC documents the doubling of filled pauses with an automated interviewer. The same work identifies mean filled pauses as the single most informative dialogue feature for predicting PTSD. These papers supply the empirical profiles we use to build counterfactuals. None of them asks what an automated evaluator does with those profiles.

The second strand concerns how models handle disfluency, and it supplies the hypothesis about type. Tian et al. report that BERT's deeper layers attend less to the discarded words, which looks like robustness. Their own numbers on 100 injected inference examples tell a different story, though. Accuracy falls from 87.50% to 84.80% under abandonment but to 80.4% under revision, so revision hurts more. Disfl-QA shows the extreme case: T5 drops from 89.59 to 61.64 F1 when questions contain corrections whose discarded words act as semantic distractors. We read these two results as consistent rather than contradictory. Models absorb benign disfluency and fail on corrections whose discarded part carries competing content. On the cleanup side, Ostendorf and Hahn's repetition model reaches 83.2 F on Switchboard for detecting the discarded span, and it targets repetitions specifically. DisAug improves detectors with LLM-generated disfluencies, but its authors note that LLMs are biased toward producing fluent text. Their method therefore requires filtering out unrepresentative generations, which constrains how far synthetic disfluency can be trusted to match real load-shaped speech.

The third strand is fairness, and it supplies the metrics and the cautions. Small Changes, Large Consequences audits LLMs in resume screening by perturbing inputs. It finds racial invariance violations up to 16.76% and up to a 72.76% exclusion rate from spacing changes alone. That is evidence that non-demographic surface noise can dominate selection, but the audit operates only on written text. Orgad et al. show that a fix applied only at the output can hide bias that resurfaces later. Goldfarb-Tarrant et al. show that intrinsic embedding metrics do not predict application bias. Together these push us to measure harm on the decisions themselves, in rankings and score shifts, rather than in representations. AHELM finds demographic unfairness in audio-language models' transcription. It also finds that a cascaded transcribe-then-reason system ranks 6th of 17, ahead of many native audio models. Its evaluation, however, uses read speech and short queries, not spontaneous disfluent dialogue. Finally, the interview-scoring work of Naim et al. and its follow-up built hiring predictors in which fillers and pauses drive the score, reaching an AUC of 0.81 against crowd judgments. The cohort consisted entirely of MIT undergraduates, and there was no fairness analysis. Taken together, this literature has measured disfluency, measured allocational unfairness, and built disfluency-sensitive evaluators. It has not asked whether evaluators respond to situationally determined disfluency composition in ways that track who speaks under load.

Core Claim

At a fixed disfluency count, a spoken-language evaluator's score shift grows with the share of revisions and abandonments relative to repetitions and filled pauses. This holds for raw LLM scoring and, at a smaller but nonzero level, for clean-then-score pipelines. Automatic cleanup is reliable on repetitions and unreliable on the hard types. Because the hard types are the ones that load and agent-led interviewing produce, the result is a disparity organized by speaking situation. The claim is falsified if type-matched and count-matched perturbations produce indistinguishable score shifts, or if clean-then-score removes the type effect entirely.

Research Program

The central artifact is a load-calibrated counterfactual set. We start from fluent answer content, then insert disfluencies using the standard three-part structure: the discarded words, an optional interruption marker such as "uh", and the repair that replaces them. The structures are not generated by prompting. They are transplanted from real annotated corpora, including Switchboard and the SCOTUS oral arguments, where repetitions make up 63% and 70% of disfluencies respectively, so that the forms are realistic. We build revisions with content-bearing discarded words following the Disfl-QA distractor recipe, for example "I led a team of five, sorry, I was a member of a team of five". We then assemble variants of each answer that hold total disfluency count fixed while varying composition. Composition targets are set from published profiles. A "low-load" profile is rich in fillers and repetitions. A "high-load" profile follows the direction Lindström et al. report, with more deletions and fewer fillers and repetitions. An "agent-interviewer" profile doubles fillers, following DAIC. Because the file reports directions rather than exact rates for the load conditions, profile magnitudes are swept rather than assumed.

The content comes from two sources. The first is cleaned transcripts of the MIT Interview Dataset, which come with existing hiring ratings. The second is spoken-style self-introductions derived from the resume benchmark of Small Changes, Large Consequences. The second source lets us cross disfluency profiles with that paper's demographic name perturbations. This crossing tests whether disfluency penalties compound with name-based bias. A model that is merely noisy should produce additive effects. A model that reads disfluency as a demographic cue should produce an interaction.

The second component is a natural experiment on real speech. DAIC interviews were conducted face-to-face, by a remote human operator, and by the autonomous agent, and each interview carries depression questionnaire scores (PHQ-9). We score the transcripts and audio with cascaded and native audio-language evaluators, prompted to estimate distress. We then measure whether estimated distress shifts with interviewer type at a fixed PHQ-9 score, and whether the false-positive and false-negative rates of a thresholded score differ by interviewer type and by gender. DAIC already reports gender-divergent nonverbal distress cues. Here, unlike in hiring, disfluency is legitimate signal, so stripping it is not an option. The fairness question becomes whether the evaluator separates disfluency caused by distress from disfluency caused by talking to a machine.

Baselines

The naive reference point is the raw pipeline: an instruction-tuned LLM evaluator receives the verbatim transcript and outputs a hiring score or a ranking over a candidate pool. The prompt and decoding settings follow Small Changes, Large Consequences, and Mixtral 8x7b is included so that its reported 16.76% racial violation rate and 72.76% spacing-driven exclusion rate serve as anchors for how large a non-demographic surface effect can be. Beating this baseline means reducing both the composition-driven score shift and the top-5 exclusion rate on the high-load profile while leaving rankings on fluent inputs unchanged.

The strongest existing competitor is clean-then-score. We run the best available disfluency remover before the same evaluator. We use two removers: an ELECTRA detector trained with DisAug augmentation, for which no reported result found on Lacuna for its Switchboard score in this file, and the sequential repetition model, which reports 83.2 F for reparandum detection on Switchboard. This is the fix most practitioners would deploy, and it is the one our hypothesis predicts will fail selectively. Both removers are trained on the same Switchboard split and given the same tuning budget. The evaluator is identical across arms. For a reference on how much gold-quality handling can recover downstream, the strongest reported figure in the file is Disfl-QA: 86.46 F1 after fine-tuning on human disfluent data, versus 82.27 with heuristic augmentation and 61.64 zero-shot. We use gold human removal on a held-out annotated subset as the upper bound for clean-then-score.

The third baseline is an evaluator fine-tuned on disfluent inputs paired with fluent-input scores, a consistency objective modeled on the Disfl-QA fine-tuning result. Its training data, size and compute are matched to the cleanup arm.

The key ablation isolates composition. We compare count-matched perturbations, consisting of random filler and repetition insertion as in standard robustness audits, against type-matched profiles at an equal disfluency count. If the high-load profile shifts scores no more than random insertion does, the type hypothesis fails.

Why not just…?

Why not just strip disfluencies before scoring? In hiring, this is a real threat. If automatic cleanup were near-perfect, it would make the composition effect disappear, and our proposal would reduce to "use a good cleaner." The available evidence suggests otherwise. The strongest reported detector number in the file comes from a model designed around repetitions. Disfl-QA shows heuristic handling of corrections stalling at 82.27 F1, against 86.46 with gold data. A remover that misses a revision leaves its distractor in place. A remover that over-deletes an abandoned phrase can cut real content, such as a claimed skill. The clean-then-score arm, with per-type breakdowns of removal recall and of how much answer content survives cleanup, settles this directly. If residual score shift after cleanup is flat across profiles, we report that cleanup suffices for hiring and refocus on the clinical setting, where stripping discards the PTSD signal DAIC identified.

Why not just fine-tune the evaluator to be invariant to disfluency? Invariance training could remove the shift on the distribution it sees. Orgad et al. show, however, that fixes applied at the output can leave the underlying sensitivity intact and let it return when the scoring layer is retrained. In clinical use, full invariance is also wrong, because some disfluency is diagnostic. We test this with the fine-tuned baseline evaluated on held-out profiles it was not trained on. We also run a restoration check: retraining only the scoring layer on unbalanced data and seeing whether the type effect returns.

Why not just run a standard metamorphic audit with random filler insertion? This is cheaper and is what existing robustness audits do. It also measures the wrong variable. Random insertion produces mostly benign fillers and repetitions, the types models already handle. The count-matched ablation is precisely this audit. If it detects the same disparity as the calibrated profiles, our calibration adds nothing and we will say so.

Why not just generate disfluent variants with an LLM? DisAug's authors report that LLMs drift toward fluent output and need filtering. Synthetic text also has no grounding in the measured load profiles. We include an LLM-generated variant set as a secondary arm and measure its type composition against the transplanted corpora. If it reproduces the same score effects, it becomes the cheaper tool for future audits.

Proposed Experiments

The first experiment is the composition sweep on MIT Interview and resume-derived content. We vary the share of hard types in steps at three fixed disfluency counts and measure the mean score shift from the fluent version, the rate of rank inversions, and the top-5 exclusion rate as defined in Small Changes, Large Consequences. We run this across the raw, cleanup, gold-cleanup and fine-tuned arms. The decisive check is the slope of score shift against hard-type share at a fixed count. We predict that it is positive for the raw pipeline, reduced but positive for automatic cleanup, and near zero for gold cleanup. The gap between automatic and gold cleanup localizes the harm in the remover's per-type recall.

The second experiment crosses disfluency profiles with the demographic name perturbations. We fit a model of score with name group, profile and their interaction. A significant interaction would show that evaluators treat disfluency differently depending on who they think is speaking. That would be a compounding of robustness failures and bias that Gap 2 of the literature leaves unexamined.

The third experiment is the DAIC natural experiment. We compute the shift in estimated distress across interviewer types at a fixed PHQ-9 score, along with false-positive and false-negative rate gaps by interviewer type and gender, for a cascaded transcribe-then-reason system and a native audio model of the kind AHELM ranks. We also compare these against a filled-pause-only predictor in the spirit of Gratch et al. The decisive check is whether moving from human to agent interviewer shifts LLM distress estimates at a fixed questionnaire score. If it does, an automated intake agent would systematically raise apparent distress for everyone it interviews. Per-gender differences in that shift would turn a calibration problem into a fairness one.

Across all three experiments, the result that would most change practice is a clean dissociation: count-matched noise leaves scores nearly untouched, while composition-matched, load-calibrated disfluency moves them, and automatic cleanup removes only the part that was already harmless.