Text-to-speech systems are starting to add "um", "ano" and stretched-out syllables so they sound like real conversation rather than audiobook narration. These hesitations carry social meaning. Listeners hear a mid-sentence "uh" combined with slow, high-pitched delivery as far less certain: on a 0–100 scale, mean certainty fell from 77 to 37 in Where's the uh, hesitation?. The models that decide where hesitations go are trained on real conversational corpora. Those corpora record how different speakers hesitate, and models trained on skewed data tend to exaggerate the skew.
This note proposes measuring and separating two effects. The first is inheritance: a synthesizer reproduces the hesitation patterns that men and women actually show in a corpus. The second is amplification: it makes the gap wider than the data. We use the Corpus of Spontaneous Japanese (CSJ), which tags every filler, repetition and prolonged phone and comes with speaker labels. We then ask a further question. When a hesitation planner trained on Japanese is applied zero-shot to English, does it carry Japanese-specific group differences into a language whose own data shows different ones?
Core Claim
Our hypothesis is that bias in spontaneous speech synthesis sits mainly in where hesitations are placed, not in how many there are. Medial fillers do the most damage to perceived confidence, and we expect learned planners to widen the gap between groups in medial-filler placement beyond what the corpus shows. If so, the obvious fix, calibrating each group's overall filler rate to the data, would leave the harmful part of the gap in place. We further expect cross-lingual transfer to carry the source language's gap into the target language, even when the target language's own data shows a different gap.
The closest prior work does not test this. Gender Bias in Instruction-Guided Speech Synthesis Models shows that Parler-TTS picks voice gender from occupation prompts, with a standardized residual of 26.6 toward male voices for "fisherman". That study covers which voice gets chosen in English read speech. It does not cover what a voice is made to do conversationally. Synthesising Uncertainty and Kirkland et al. establish that disfluencies change perceived certainty, but they use single-speaker English stimuli and never compare groups. Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer shows that the direction of cross-lingual alignment changes text bias: aligning English into Spanish reduced intrinsic bias by 23%. Speech and Japanese are both untested there.
Measuring amplification against the data
We borrow the main lesson of Directional Bias Amplification: apparent amplification has to be measured against base rates in the data and split by direction. That paper found that recalibrating decision thresholds reduced measured amplification from 0.040 to 0.001. For a hesitation type $h$ (initial filler, medial filler, prolongation), a speaker group $g$, and a discourse context $c$ (clause position, talk type, preceding word class), let $r_h(g,c)$ be the rate at which $h$ occurs. We define the group gap
$$\Delta_h = \mathbb{E}_c\left[r_h(F,c) - r_h(M,c)\right],$$
where the average runs over contexts weighted by their corpus frequency. Amplification in the attribute-to-behaviour direction is
$$A_h = \Delta_h^{\text{model}} - \Delta_h^{\text{data}},$$
where $\Delta_h^{\text{model}}$ is the gap in the model's hesitation plans for held-out CSJ transcripts with fillers removed, and $\Delta_h^{\text{data}}$ is the gap in the original annotations. Conditioning on context matters. Without it, a model would look biased simply because women and men in CSJ give different kinds of talks.
We also measure the other direction, from behaviour to attribute. We train a probe on hesitation plans to predict speaker group, and compare its accuracy on model plans with its accuracy on real annotations. Generated plans that reveal gender more easily than real speech do are a form of amplification that a planner can show even with no explicit group input.
Research Program
We build a two-stage synthesizer. The first stage is a hesitation planner: a multilingual LLM fine-tuned with LoRA adapters, following the FillerSpeech recipe. It reads disfluency-free text and predicts the position, type, duration and pitch of each filler and prolongation. The planner is trained on CSJ transcripts, with targets drawn from the disfluency and prolongation tags documented in DNN-based Speech Synthesis Using Abundant Tags of Spontaneous Speech Corpus. It is trained in two variants: one conditioned on speaker group and one with no group input.
The second stage is an acoustic model that turns the planned tags into speech. For Japanese we use the extended-context tag-conditioned model of Yamashita et al., trained on CSJ. For English we use a multilingual flow-matching generator in the style of Voicebox. A flow-matching generator learns to transform noise into speech along a smooth path, and it accepts inserted filler tokens.
Splitting the synthesizer into stages lets us place the bias. We can ask whether it arises in the text-level plan, in the pitch and duration the planner assigns to each filler, or in the acoustic rendering.
For the cross-lingual part, we run the CSJ-trained planner zero-shot on English Switchboard transcripts with disfluencies removed. We compare its group gaps with Switchboard's own annotated base rates, using the speaker demographics that Switchboard-Affect relies on. English is chosen as the target because we have ground truth for it, which lets us separate "imported the source language's gap" from "matched the target language's gap". The practical setting the work is meant to inform is low-resource target languages, where no such check exists.
We first report how often fillers and backchannels can be told apart in the annotations. Stolcke and Droppo show that ASR systems confuse them. If the reference labels themselves are noisy, the base rates we compare against are noisy too.
Baselines
The do-nothing reference is the corpus itself, played back. Annotated CSJ and Switchboard hesitation patterns, rendered by the same acoustic model, define $\Delta^{\text{data}}$. Every planner is judged against them. Beating this reference does not mean scoring higher. It means a planner whose amplification $A_h$ has a confidence interval that includes zero while staying natural.
The naive placement baseline puts fillers at clause boundaries according to the per-group frequencies observed in CSJ. Rule-based insertion of this kind once improved naturalness (Marge et al.). By construction it cannot amplify anything beyond the corpus marginals, so it shows what a "fair but crude" system costs in naturalness.
The strongest existing method is the FillerSpeech-style LLM planner, which outperformed random insertion at predicting filler position, type, duration and pitch. We found no reported numerical MOS result for FillerSpeech on Lacuna. For the CSJ acoustic stage, the reference figure is the Yamashita et al. XAB preference: 55.7% versus 44.3% for speaker F1 ($p<0.01$), and a non-significant 53.4% for M2. Our Japanese rendering should match this preference against baseline context. All planners share the same base LLM, training split, adapter rank and tuning budget, and are selected on naturalness on a held-out set, never on bias.
Three ablations isolate the key ingredient. The first applies rate calibration after the fact: per-group thresholds that bring overall filler rates to the corpus base rates, following Wang and Russakovsky. This tests whether matching how many fillers each group gets is enough. The second removes group conditioning. The third is a position-aware calibration that matches base rates separately for initial and medial slots. Our claim predicts that the third removes the perceived-certainty gap and the first does not.
Why not just…?
Why not just calibrate filler rates per group and be done? Wang and Russakovsky's calibration result makes this the most serious threat. If matching overall rates removes both $A_h$ for medial fillers and the gap in listener certainty ratings, our central claim is wrong, and the paper becomes a report that a simple fix works. The rate-calibration ablation is designed to settle exactly this. Our reason to doubt it is Kirkland et al.'s finding that filler position interacts with speech rate and pitch, so equal counts can still produce unequal perceived certainty.
Why not just remove speaker gender from the planner's input? Zhao et al. found that scrubbing gendered words did not remove bias from representations. Wang and Russakovsky found that masking the attribute increased behaviour-to-attribute leakage, because the model fell back on contextual proxies. The no-group-conditioning variant tests the speech analogue. We measure whether probe accuracy on its plans drops to the level on real data or rises above it, which would mean topic and word choice are acting as proxies for gender.
Why not just use random or rule-based fillers, which cannot amplify? They are fair only in a trivial sense. Dall et al. show that listeners have consistent preferences for filler placement (Dall et al.), and the FillerSpeech results show that learned placement beats random. The naive baseline measures how much naturalness a bias-free planner gives up. If that cost turns out to be small, it is a legitimate recommendation, and we would report it.
Why not translate and use a native English planner instead of zero-shot transfer? For English this would work, and we use exactly that as the ground truth. The cross-lingual experiment is not offered as a better English system. It measures whether transfer imports source-language group gaps, which is the risk faced in languages without spontaneous corpora. MoonCast also reports that removing spontaneous tokens lowers perceived quality, so a cascade that drops source hesitations is not equivalent anyway.
Proposed Experiments
Experiment one estimates $\Delta_h^{\text{data}}$ and $A_h$ on CSJ for each hesitation type, with bootstrap confidence intervals resampled over speakers. Experiment two applies the three ablations and recomputes $A_h$ and probe leakage.
Experiment three is a listening test using the 0–100 certainty scale of Kirkland et al. The key control is that every plan is rendered in one fixed voice and also in voices of both genders. With the voice fixed, any certainty gap must come from the hesitation plan alone. Kuan and Lee's voice-selection bias is not in play because we choose the voice ourselves.
Experiment four applies the CSJ-trained planner to English and compares the transferred gap with both the CSJ gap and the Switchboard gap. We call the result norm import if the transferred gap is closer to the CSJ gap.
The decisive check is a conjunction of three conditions. Medial-filler amplification must be significantly positive. It must survive rate calibration. It must produce a measurable listener certainty gap under a fixed voice. If amplification is within noise of zero, the model only inherits corpus behaviour. In that case the contribution becomes the measurement tool together with an inheritance audit of CSJ and Switchboard.
References
- Where's the uh, hesitation? The interplay between filled pause location, speech rate and fundamental frequency in perception of confidence
- Gender Bias in Instruction-Guided Speech Synthesis Models
- Synthesising Uncertainty: The Interplay of Vocal Effort and Hesitation Disfluencies
- Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer
- Directional Bias Amplification
- FillerSpeech: Towards Human-Like Text-to-Speech Synthesis with Filler Insertion and Filler Style Control
- DNN-based Speech Synthesis Using Abundant Tags of Spontaneous Speech Corpus
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
- Switchboard-Affect: Emotion Perception Labels from Conversational Speech
- Comparing Human and Machine Errors in Conversational Speech Transcription
- Towards Improving the Naturalness of Social Conversations with Dialogue Systems
- Investigating automatic & human filled pause insertion for speech synthesis
- MoonCast: High-Quality Zero-Shot Podcast Generation