Stabilized Speech Continua: Realistic One-Cue Stimuli With a Built-In Stability Check

Speech scientists build acoustic continua to find out which cues listeners use. A continuum is a series of tokens that differ along one dimension, such as pitch height or the amount of creaky voice. The standard tools are signal-processing edits like PSOLA, which shifts pitch by overlapping and re-adding waveform chunks, and amplitude modulation of alternate glottal cycles to simulate creak. These tools move the target cue and leave everything else frozen. Natural speech does not behave that way. When a speaker creaks, the spectrum, the pulse timing, and the formants all shift together. Frozen stimuli can therefore produce misleading null results. The clearest case is Korean: speakers produce creak on fortis stops, yet artificial creak had no significant effect on fortis/lenis judgments, and the authors themselves suspect the manipulation rather than the cue (The function of creaky voice in South Korean). Synthetic speech also costs listeners more effort even when they understand it. Pupil dilation rises although word recall stays at or above 91% (Using Pupillometry to Measure the Cognitive Load of Synthetic Speech), and listeners converge phonetically toward natural speech but barely toward synthetic speech (Shadowing Synthesized Speech).

This proposal borrows one procedure from string phenomenology. In that field, some scalar fields ("moduli", which set the size and shape of hidden dimensions) are pinned by an external constraint. The remaining fields are then allowed to settle into a minimum of the potential energy. Every candidate ground state is checked for two things: it must have no unstable directions, and the energy must not run away. We apply the same procedure to speech stimuli. The experimenter pins the target cue to a value. A learned model of natural speech then lets every other acoustic dimension settle to its most natural value given that cue. Each token is released only if it passes a stability check. The result is a continuum that is controlled in exactly one coordinate and natural in all the others, with a per-token certificate saying so.

Core Claim

The claim is that stimulus realism can be made operational, not just rated. A token counts as realistic in this sense if two conditions hold. First, it sits at a local minimum of a learned natural-speech energy, subject to the experimenter's constraint on the cue. Second, that minimum is well inside the range of energies seen on real recordings, along the whole continuum and not only at its endpoints. We expect such continua to recover perceptual effects that frozen continua miss, to lower listening effort, and to restore phonetic convergence.

Let $E(x)$ be a learned energy over mel-spectrogram frames $x$, where low energy means likely natural speech. Let $c(x)$ be a differentiable estimator of the target cue, for example mean $F_0$ over the vowel or a creak index. Each continuum step $t$ is defined as

$$x^*(t) = \arg\min_x E(x) \quad \text{subject to} \quad c(x) = t,$$

with the search initialized from a natural recording.

The first condition is the analogue of "no tachyons", i.e. no directions in which the energy curves downward. It requires that the Hessian of $E$, restricted to directions that keep $c$ fixed, has only positive eigenvalues:

$$\lambda_{\min}\big(P_t^\top \nabla^2 E(x^*(t))\, P_t\big) > 0.$$

Here $P_t$ is an orthonormal basis of the tangent space of the constraint surface $c(x)=t$, meaning the set of small edits that leave the cue unchanged.

The second condition is the analogue of boundedness. It requires $E(x^*(t))$ to stay below the 95th percentile of energies measured on held-out natural recordings for every $t$.

A negative eigenvalue has a concrete meaning. There is a cue-preserving edit that would make the token more natural. In practice this is where "buzz", category drift, or identity drift is waiting to happen.

Relation to Prior Work

The physics papers supply the procedure. They are a methodological lens, not a physical claim. Three strands matter. In Flux compactification, background fluxes fix some moduli while others need a separate mechanism. This is the pattern of "pin some coordinates by constraint, let the rest be fixed by dynamics" that our formulation copies. Building a Better Racetrack locates critical points numerically and then verifies that every modulus mass is positive, which rules out tachyons. We reuse that practice of certifying each candidate point by its Hessian spectrum rather than trusting the optimizer. On the Boundedness of Effective Potentials Arising from String Compactifications proves that the potential cannot fall to minus infinity. Our boundedness condition is a far weaker empirical stand-in, a quantile check rather than a proof, and we say so.

Two papers motivate checking along the whole path and reading instabilities as category changes. Novel theoretical constraints for color-octet scalar models requires stability at every energy scale up to 63 TeV, not only at the electroweak scale. By analogy, we check every continuum step. Constraining the vacuum expectation values in naturally R-parity conserving supersymmetric models shows that a negative eigenvalue in a mass matrix makes the ground state break a symmetry, electric charge or color. Our speech analogue is a token whose unstable direction leads toward a different phonetic category.

On the speech side, two papers supply the machinery. Energy-Based Models For Speech Synthesis trains an explicit speech energy and refines outputs with Langevin sampling, a noisy gradient descent on that energy. EdiTTS edits pitch through a score model, which learns the gradient of log-density, without retraining. Neither paper constrains a named cue exactly while relaxing the rest, and neither checks curvature. Both evaluate with MOS or MCD rather than perception experiments.

Several papers show the problem our method targets: moving one control disturbs the others. Controlling formant frequencies with neural TTS reports artifacts when formants and pitch are moved together. Hierarchical Generative Modeling for Controllable Speech Synthesis and Learning Robust Latent Representations for Controllable Speech Synthesis document latent entanglement.

Research Program

We would take a pretrained score-based acoustic model in the Grad-TTS/EdiTTS family and treat its score as $-\nabla E$. The Jacobian of the score is then the negative Hessian. Projected Hessian-vector products are cheap, so we estimate $\lambda_{\min}$ with a few dozen Lanczos iterations (an iterative method for extreme eigenvalues of large matrices) instead of forming the full matrix.

The cue estimators $c$ come from existing tools where they exist. For $F_0$ we use a differentiable pitch tracker, and for formants the formant predictor of the FastPitch variant. For creak, where no such estimator exists, we train a creak regressor on natural glottalized versus modal vowels.

Constrained minimization runs as projected Langevin steps with annealed noise, followed by a deterministic polish. Tokens that fail either condition are re-initialized from a different natural recording. If failures persist, that step of the continuum is reported as unreachable rather than silently kept.

For every token, the pipeline logs which secondary cues moved and by how much, such as spectral tilt, H1–H2, jitter, and duration. Relaxation means secondary cues vary, and an experimenter must be able to model that variation as a covariate. This is the main honest cost of the approach relative to strict one-cue isolation.

Baselines

The naive reference point is the field's current practice. For pitch this means Praat PSOLA continua, as in Perceptual Learning in Lexical Tone. For creak it means alternate-cycle amplitude modulation, as in The function of creaky voice in South Korean. These are run on the same natural base recordings and the same continuum endpoints, measured with the same cue estimator. They define the frozen-complement condition any realism claim must beat. A signal-processing reference comes from EdiTTS, where WORLD-vocoder pitch shifting scored MOS 2.79 on LJSpeech.

The strongest existing method is EdiTTS score-based pitch editing, which already relaxes spectral detail through its diffusion prior. On LJSpeech it reports an upward pitch-shift MOS of 4.04, against 3.77 for FastPitch (EdiTTS). We run it on the same base model and utterances, with the same total number of reverse steps as our constrained sampler. Beating it requires two results. The first is MOS at least equal to EdiTTS within confidence intervals, with lower cue error (achieved $c$ minus target). The second, which is the real test, is a significantly larger perceptual effect or lower peak pupil dilation in the human studies below. If we match EdiTTS on MOS but not on those human measures, the method adds nothing beyond certification.

A FastPitch-formant-predictor baseline (Controlling formant frequencies) represents fast non-iterative control. For it, we report real-time factor alongside quality.

Two ablations isolate the key ingredients. In the first, constrained minimization is kept but the curvature check is removed and only the energy quantile is applied. This tests whether the Hessian certificate is doing any work. In the second, the target cue is pinned but the complement is frozen at the base recording's values, with only local smoothing. This isolates relaxation itself.

Why not just…?

Why not just use PSOLA or WORLD, which give exact deterministic control? Control is not the problem; the absence of natural covariation is. WORLD pitch shifting scored MOS 2.79 against 4.04 for score editing (EdiTTS). Amplitude-modulated creak produced a null perceptual result that clashes with production data (Korean creak study). The Korean replication below decides this directly. If PSOLA-style and stabilized creak continua give the same identification slopes, the frozen approach was adequate and our claim fails.

Why not just use EdiTTS-style score editing, which already relaxes the rest? This is the real threat. EdiTTS may already produce most of the realism, in which case our contribution shrinks to exact cue pinning plus a diagnostic. The head-to-head on cue error, perceptual slope, and pupil dilation settles it. We commit in advance to reporting the curvature-free ablation. If certified and uncertified tokens are indistinguishable perceptually, we will drop the curvature check and present the method as constrained score editing.

Why not just record natural tokens spanning the continuum, or cross-splice natural segments? Natural recordings cannot hold other cues fixed while one varies. Splicing introduces its own confounds. Naive spectrogram concatenation raised WER to 11.61% against 9.11% for score-based refinement (EdiTTS). The Korean splicing stimuli showed a large word effect, with an odds ratio of 2.47 for /tadzut/ versus /tasja/ (Korean creak study). We keep a cross-spliced condition in the creak experiment so that this argument is tested, not assumed.

Why not just filter generated tokens with an automatic MOS predictor or a likelihood threshold? A scalar threshold is exactly the energy-only ablation. It detects tokens that look unnatural in aggregate. It does not detect tokens that sit on a slope where a cue-preserving edit would drift toward another category. Whether that difference matters perceptually is an empirical question, and the ablation answers it.

Proposed Experiments

Objective quality. The first stage uses LJSpeech and VCTK. We report MOS, WER, cue error, speaker-embedding similarity to the base recording, and the fraction of continuum steps certified. VCTK tests whether certification rates hold across 108–110 speakers and accents.

Listening effort. The second stage uses Harvard sentences in noise at −1, −3 and −5 dB SNR, following Impact of Different Speech Types on Listening Effort, with 500 Hz pupillometry as in Using Pupillometry. We compare natural speech, EdiTTS-edited, stabilized-edited, and PSOLA-edited versions of the same sentences with matched pitch shifts. Neither file source gives absolute pupil dilation values (no reported result found on Lacuna for peak pupil dilation magnitudes). Our success criterion is therefore relative: the stabilized tokens should show peak pupil dilation not significantly different from natural speech, while PSOLA tokens remain elevated.

The decisive checks. The third stage consists of two human experiments that reproduce published paradigms.

The first reruns the Korean fortis/lenis ABX task with VOT fixed at 40 ms, crossing creak manipulation (amplitude-modulated, cross-spliced, stabilized) with the lexical items. This requires collecting a small Korean read-speech corpus to adapt the model; no Korean training data appears in the file. The prediction is a significant creak effect only in the stabilized condition, once logged secondary-cue changes are entered as covariates.

The second reruns the Mandarin exposure-test recalibration with Tone 1 to Tone 2/Tone 3-sandhi continua (Perceptual Learning in Lexical Tone). The original found recalibration at $p<0.001$ and cross-category generalization, with $p=0.212$ for adding tone category to the model. The prediction is that stabilized continua preserve recalibration but change how it generalizes. That outcome would tell us whether "phonetic, not categorical" learning partly reflects the frozen stimuli.

A German shadowing replication (Shadowing Synthesized Speech) is a secondary check. It asks whether stabilized tokens restore [e:]/[ɛ:] convergence to natural-speech levels.

A null in the Korean experiment, together with no pupil-dilation advantage over EdiTTS, would falsify the central claim.