Voice-Cloned Disfluency Phantoms for Calibrating Speech Monitoring During Brain-Tumor Therapy

Some brain-tumor treatments only work during a short, unpredictable stretch of time. Low-dose anti-angiogenic drugs, which act on the tumor's blood supply, can briefly repair a tumor's leaky, chaotic blood vessels. During that period, pressure and swelling in the tumor drop and immune drugs penetrate better. The period then closes again. Clinicians cannot see it opening without a scan or a biopsy, and scans are too sparse to catch it. Speech is the cheapest signal a patient produces every day, and it changes with brain function: hesitations, prolonged sounds and unsteady pitch rise with neurological load. The problem is that nobody knows how small a change in a given patient's speech a monitoring system can reliably detect against that person's ordinary day-to-day variation. Without that number, a speech readout cannot be trusted to flag anything.

We propose to measure that sensitivity with synthetic "phantoms," borrowing the term from medical imaging, where a phantom is a test object with known properties used to calibrate a scanner. A spontaneous text-to-speech system clones the patient's own voice from a baseline recording. It then generates conversational speech with a known, graded dose of disfluency (filled pauses, prolongations, blocks, pitch instability) added. Running any speech-monitoring pipeline on these phantoms gives a per-patient minimum detectable change and an expected time-to-detection. Both can be checked against real disfluent speech before anyone claims the method works on a clinical cohort. Spontaneous speech synthesis is the core mechanism here. Vascular normalization is the application that sets the requirements: detection has to be fast, has to work for one person, and has to rest on few samples.

Core Claim

Modern spontaneous speech synthesizers can already place disfluencies at controllable rates in a cloned voice. That makes them usable as calibration instruments, not just as data generators. The claim is that the sensitivity a detector shows on a speaker's voice-cloned phantoms predicts its sensitivity on that speaker's real disfluent speech. If so, the same procedure can certify whether a speech biomarker is sensitive enough to resolve a treatment window lasting days or weeks.

We are deliberately not claiming that speech measures blood vessels. The glioblastoma imaging study (Therapy-induced modulation of tumor vasculature and oxygenation) found that bevacizumab normalized vessel diameter in a mouse model yet failed to relieve hypoxia, while temozolomide reoxygenated the tissue. That result motivates and constrains this work. Structural readouts and functional state come apart. A functional readout such as speech is therefore worth having, but it measures neurological function and not vessel geometry. It is meant to tell clinicians when to order a scan, not to replace one.

What exists and what is missing

On the synthesis side, several systems already provide the knobs a phantom needs. Controllable Context-Aware Conversational Speech Synthesis predicts where disfluencies go with a BERT model and scales spontaneity with a frequency threshold. FillerSpeech controls the pitch and style of filled pauses but reports that the added pitch predictor hurts pronunciation. Spontaneous Style TTS with Controllable Spontaneous Behaviors models 19 behavior types on a VALL-E backbone. VibeVoice (VibeVoice) and MoonCast (MoonCast) produce long, zero-shot cloned conversational audio.

All of these systems aim to sound natural. None asks whether the injected dose is faithful, meaning whether "10% more prolongations" in the control actually yields that much measurable change and nothing else. Two findings show this is a real concern. Linguistic speech regularization with pseudo-filled-pause insertion showed that inserting fillers degrades the neighboring words unless an extra regularization loss is added. VibeVoice found that too many diffusion denoising steps strip speaker texture. For a phantom, either side effect is a contaminant, because it changes the signal by something other than the dose.

On the detection side, Synthetic Data Generation in Speech Dysfluency Detection built LLM-Dys, a 12,790-hour synthetic corpus, and trained detectors on it. That work uses synthesis to train classifiers of disfluency events. We instead use synthesis to measure a fixed detector's sensitivity to graded change within one voice, which their work does not do. SEP-28k supplies the main obstacle: disfluency is highly variable with stress and context. Any threshold has to be set against a speaker's own noise and not a population norm. On the oncology side, the normalization window is known to be transient, but there is no non-invasive way to track it. That gap is why a dense, cheap, individually calibrated readout matters.

Research Program

The first component is the phantom generator. We take a zero-shot spontaneous synthesizer and expose a continuous dose vector over disfluency rate per class (filled pause, prolongation, block, repetition) and pitch jitter. Each phantom is generated from the speaker's baseline audio and a fixed conversational script. Phantom fidelity is enforced in two ways. First, we add the feature-matching regularizer from the pseudo-filled-pause work so that the words around a filler stay unchanged. Second, we choose a small number of diffusion steps that preserves speaker similarity.

Fidelity is then audited directly. An independent, forced-alignment-based measurement must recover the injected dose with a linear slope near one. The non-filler words must also show no shift in their acoustic features. Phantoms that fail this audit are discarded.

The second component is the calibration statistic. For speaker $s$, detector $D$ and dose $\delta$, let $p_s(\delta)$ be the probability that $D$ flags a phantom session at dose $\delta$. The threshold is set so that $D$ raises a false alarm on 5% of that speaker's unperturbed sessions. These unperturbed sessions are resampled real recordings plus zero-dose phantoms, so the null distribution reflects the speaker's own variability. The minimum detectable change is

$$\delta^*_s = \min \{\delta : p_s(\delta) \ge 0.8\},$$

which is the smallest dose detected with 80% power at a 5% false-alarm rate. Combining $\delta^*_s$ with an assumed sampling frequency, for example one short recording per day, gives an expected detection delay for a change of a given size. That delay can be compared with how long a normalization window lasts.

The third component is validation against real speech. The phantom is only useful if $\delta^*_s$ measured on synthetic audio predicts detection on real disfluent audio from the same speaker.

Baselines

The naive reference point is a population-normed detector with no personalization. Each session's disfluency rate is converted to a z-score against corpus-wide statistics, and a session is flagged above a fixed cutoff. This is what most clinical speech-biomarker pipelines implicitly do. Beating it on per-speaker false-alarm control is the minimum bar.

The stronger personalized baseline sets thresholds only from the speaker's own unperturbed sessions, using standard acoustic and disfluency features with no synthetic data. It is the most serious competitor, discussed further below. It gets the same recordings, the same feature extractor and the same false-alarm target.

The strongest existing detector is the Whisper-large-v3-turbo model trained on LLM-Dys. That paper reports a word-level F1 of 0.977 on UCLASS (Synthetic Data Generation in Speech Dysfluency Detection). On FluencyBank, the improved ConvLSTM in SEP-28k reports an F1 of 83.6 and an equal error rate of 16.9. We use both as detectors $D$ inside our protocol, retrained on the same data split with the same tuning budget.

The question here is not whether we beat their event F1. It is whether phantom calibration yields thresholds that hold their nominal 5% false-alarm rate and 80% power on real held-out speech from each speaker, where uncalibrated thresholds on the same detector do not. Two ablations isolate the key ingredient. In the first, phantoms use a generic voice instead of the speaker's clone, which tests whether personalization matters. In the second, phantoms use LLM-prompted disfluency insertion instead of the continuous dose knob, which tests whether known graded doses matter.

Why not just…?

Why not just set thresholds from the patient's own pre-treatment recordings? This is the real threat. Baseline variance tells you the false-alarm side, but it says nothing about power: how large a true neurological change must be before this detector, on this voice, notices it. Phantoms provide labeled positives in the same voice. The comparison is direct. If personal-baseline thresholds already reach nominal power on real held-out disfluent clips across SEP-28k speakers, the phantom adds nothing, and we will report that outcome.

Why not just train detectors on synthetic dysfluency, as LLM-Dys does? Training improves event classification. The same paper also reports a scaling plateau around 36,000 samples, and its scores are population-level. A better classifier still lacks a per-speaker sensitivity curve, so our protocol treats any trained detector as an input and calibrates it rather than competing with it.

Why not just prompt an LLM to add disfluencies, as MoonCast does? MoonCast shows that prompting Gemini raises spontaneity by +0.85 in English. Prompting does not produce a known, monotone dose, and the dose is what calibration needs. The prompting ablation tests this: if prompt-based phantoms predict real sensitivity as well as dose-controlled ones, we drop the control machinery.

Why not just use MRI to find the window? For measuring vessels, MRI is the right tool, and speech cannot replace it. The argument for speech is temporal density. We found no reported result on Lacuna linking speech changes to edema or vascular normalization in any cohort. That link is therefore the unproven step, and it is reserved for a pilot rather than assumed.

Proposed Experiments

Experiment 1: phantom fidelity. We use DisfluencySpeech (DisfluencySpeech) and the speakers in SEP-28k. We generate dose sweeps and test dose recovery with forced alignment. We also measure character error rate on the non-filler words, using the 4.04% ground-truth figure from DisfluencySpeech as the reference. Finally, listeners do a forced-choice test between phantom and real disfluent clips.

Experiment 2: the decisive check. We compute $\delta^*_s$ from phantoms for each recurring SEP-28k and FluencyBank speaker, for each candidate detector. We then measure the realized power and false-alarm rate on that speaker's held-out real clips. The hypothesis is supported if two things hold. Phantom-derived thresholds should achieve close to 80% power and 5% false alarms on real clips. The ranking of detectors by phantom $\delta^*$ should also match their ranking by real-data sensitivity, assessed with a Spearman correlation across detectors and speakers. Both must exceed the personal-baseline and generic-voice ablations. Failing the first condition means synthetic and real disfluency differ in ways detectors notice, and the clinical step should not proceed.

Experiment 3: clinical feasibility pilot. This is only a design. Patients with glioblastoma receiving anti-angiogenic therapy would record two minutes of spontaneous speech daily, alongside their scheduled MRI. Experiment 2's $\delta^*$ would determine whether daily sampling could detect changes within a window's duration before enrollment begins. No reported result found on Lacuna for speech correlates of vascular normalization. The pilot therefore tests feasibility and power, not efficacy.