Do Speakers Shift Style in Their Own Way? A Random-Slope Model of Speaker Embeddings Across Read and Conversational Speech

Speaker verification is the task of deciding whether two recordings come from the same person. These systems fail badly when one recording is read aloud and the other is casual conversation. On the UCLA Speaker Variability Database, an x-vector system (a neural network that maps an utterance to a fixed-length "voiceprint" vector) scores 14.35% equal error rate when both sides are read speech. The rate rises to 31.46% when read speech is compared with telephone conversation (Speaker Discrimination in Humans and Machines: Effects of Speaking Style Variability). Human listeners degrade less, from 6.96% to 20.68%. The field treats this style shift as generic noise to be averaged away or removed. We propose to model it with the tool statisticians built for exactly this kind of nested data: a mixed-effects model. Such a model splits variation into a population-wide part and a per-individual part. Here the per-individual part is a random slope: each speaker gets their own style shift, and that shift is allowed to correlate with where the speaker's voice sits in embedding space.

The core claim can be tested and could come out false. Moving from read to conversational speech changes a speaker's embedding in three ways. The first is a shift common to everyone. The second is a shift that depends predictably on who the speaker is. The third is idiosyncratic and unpredictable. Only the first two can be corrected at test time for a speaker never seen speaking casually. The relative sizes of the three parts decide whether style-robust verification from one-style enrollment is possible at all. A model that estimates these parts with partial pooling across speakers can do two things. It can tell us how large each part is, which answers the open phonetic question of whether spontaneous-speech reduction is uniform across people or speaker-specific. It can also give a scoring rule that "predicts" a new speaker's conversational voiceprint from their read enrollment.

Core Claim

Let $x_{sk}$ be a frozen speaker embedding of an utterance by speaker $s$ in style $k$ (read, conversation, and in the richer corpora also interview, oration, or vocal-effort levels). We fit

$$x_{sk} = \mu + \beta_k + (I + \Gamma_k)\, y_s + w_{sk} + \epsilon, \qquad y_s \sim \mathcal{N}(0,\Sigma_y),\; w_{sk} \sim \mathcal{N}(0,\Sigma_w),\; \epsilon \sim \mathcal{N}(0,\Sigma_{\epsilon,k}).$$

The terms mean the following. $\mu$ is the global mean. $\beta_k$ is a fixed style offset shared by all speakers. $y_s$ is the speaker's latent identity. $\Gamma_k$ is a low-rank matrix, set to zero for the reference style. It lets the style shift depend on the speaker, which is the random slope correlated with the random intercept. $w_{sk}$ is a speaker-by-style interaction that no speaker property predicts. $\epsilon$ is utterance noise whose spread differs by style.

With $\Gamma_k = 0$ and no $w$, the model collapses to a style-offset version of PLDA, the probabilistic linear discriminant analysis that standard x-vector systems already use for scoring. With $\beta_k = 0$ as well, it is plain PLDA. Verification scores come from the likelihood ratio under the joint Gaussian of enrollment and test embeddings. The cross-style covariance is $(I+\Gamma_r)\Sigma_y(I+\Gamma_c)^\top$, and $\Sigma_w$ inflates only the mismatched pairs.

Our prediction is that $\Gamma_k$ carries real, held-out-verifiable structure. The reasoning is that reduction in conversational speech acts on vowel space and spectral shape, as shown for spectral compression (Analysis of spectral space reduction in spontaneous speech and its effects on speech recognition performances) and for style effects that stay separate from rate (Modeling Formant Dynamics in Mandarin /ai/: Effects of Speech Style and Speech Rate). How much a voice compresses should depend on where it starts, for example on its vowel-space extent and source spectral shape. If instead $\Gamma_k$ adds nothing over $\beta_k$, the right answer is the simpler fixed-offset model, and we will report that.

Relation to Prior Work

Two lines of work motivate the proposal. Afshan et al. document the mismatch collapse but score with standard PLDA, treating style as an unmodeled condition. Ferragne et al. show on SRI-FRTIV that vocal-effort mismatch hurts a GMM-UBM verifier more than style mismatch, and that interview and conversation behave interchangeably (Effects of vocal effort and speaking style on text-independent speaker verification). This is a useful constraint: not every style label deserves its own slope, and the model should learn that some $\Gamma_k$ are near zero.

On the modeling side, linear mixed models with nested random effects already explain within-speaker tempo variation, where demographic tempo differences vanish once phrase length is modeled (Modeling of between-speaker and within-speaker variation in spontaneous speech tempo). GAMMs, which are mixed models with smooth nonlinear terms, have been used to separate style from rate in formant tracks. Both work on hand-measured acoustic quantities, and neither asks whether style effects vary by speaker inside a learned representation.

Two papers supply the method but apply it elsewhere. Mixed Effects Neural Networks put a fixed-plus-random-effects layer on a ResNet for gaze estimation and learn to predict random offsets for new subjects (Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation). Neural Mixed Effects splits network weights into shared and person-specific parts with Gaussian shrinkage (Neural Mixed Effects for Nonlinear Personalized Predictions). Both personalize a predictor for one grouping factor. Neither crosses two factors (speaker and style), and neither studies the covariance between a person's intercept and their condition slope. That covariance is exactly what allows zero-shot correction for unseen speakers.

We deliberately keep the mixed-effects part linear and on top of frozen embeddings. The corpora with controlled style contrasts are small: 40 to 100 speakers in UCLA, 30 in SRI-FRTIV. At that scale a low-rank Gaussian model is what the data can support, and it keeps the variance decomposition readable.

Research Program

We first extract embeddings with a fixed pretrained x-vector extractor comparable to the one in Afshan et al., and separately with a self-supervised wav2vec 2.0 encoder (mean-pooled). This tests whether the finding depends on the representation. The model is fit by expectation-maximization with $\Gamma_k$ of rank 1 to 8, chosen by held-out speaker likelihood.

The scientific output is a decomposition of the style-induced change in embeddings into three fractions: the common offset, the part predicted by the speaker, and the idiosyncratic interaction. Each fraction comes with a speaker-level bootstrap interval. We use a bootstrap because normality of random effects is doubtful for speech (Simple bootstrap for linear mixed effects under model misspecification).

To connect the embedding result to phonetics, we fit the same crossed model to interpretable acoustic measures on the same utterances. These are vowel-space dispersion and the harmonic spectral-shape and energy measures that account for most voice variance in Linguistic versus biological factors governing acoustic voice variation. We then ask whether speakers with large random slopes in embedding space are the ones whose vowel space compresses most. This gives the embedding-level effect a physical interpretation, or shows that it has none.

Baselines

The naive reference is the off-the-shelf x-vector/PLDA system scored with no style information. On UCLA read-versus-conversation trials, the reported equal error rate is 31.46% in Speaker Discrimination in Humans and Machines: Effects of Speaking Style Variability. We will reproduce this on our speaker folds, so our comparison uses our own number and the published figure only as a sanity check.

The strongest cheap competitor is fixed-offset PLDA. It subtracts a per-style mean and gives each style its own noise covariance. This is our model with $\Gamma_k = 0$ and $\Sigma_w = 0$. A second strong competitor is PLDA retrained on pooled multi-style training speakers. Both use the same embeddings, the same training speakers, and the same speaker-disjoint cross-validation folds. They receive the same hyperparameter search budget (rank and regularization) as our model.

The key ablation removes only $\Gamma_k$ while keeping $\Sigma_w$. This isolates whether speaker-dependent slopes help beyond an unpredictable interaction term.

The strongest learned alternative is adversarial style removal with a gradient reversal layer, as in Adversarial Speaker Verification. That paper reduced unseen-condition error from 13.02% to 11.13% for noise, but there is no reported result on Lacuna for adversarial training on UCLA or SRI-FRTIV style mismatch. We will train a small projection head on the same frozen embeddings with the reversal loss, tuned with the same budget.

The human-listener mismatch rate of 20.68% serves as an external reference point. "Beating" the baselines means a statistically reliable reduction in mismatched-style error against fixed-offset PLDA, measured by a speaker-level bootstrap, with matched-style error not worse. Closing any meaningful part of the gap to the human figure would be a strong result.

Why not just…?

Why not just subtract a per-style mean from the embeddings? This is the real threat. If the style effect is mostly a shared offset, it will match our model at a fraction of the complexity. We do not argue it away. The $\Gamma_k$ ablation measures it directly. If held-out likelihood and mismatched-trial error show no gain from random slopes, the paper's conclusion becomes "style shift is uniform across speakers in this embedding." That is itself a scientific answer to Gap 3. The accompanying fraction estimates would still say how much error is irreducibly idiosyncratic.

Why not just train adversarially to make embeddings style-invariant? Adversarial training removes whatever predicts style. Here it faces a specific problem: if speakers' style shifts correlate with their identity, which is our hypothesis, then stripping style information also strips speaker information. Our model instead keeps and uses that covariance. The adversarial baseline, run on identical data, will show which effect dominates. The file has no evidence of adversarial methods on style mismatch, so this is genuinely open.

Why not just use a bigger pretrained encoder or fine-tune on multi-style data? Large pretrained representations help downstream tasks, such as wav2vec 2.0 with BERT over hand-crafted features in The 1st SpeechWellness Challenge: Detecting Suicide Risk Among Adolescents. But the x-vector system in Afshan et al. was trained on large, varied data and still doubled its error under mismatch. With only tens of style-contrasted speakers, fine-tuning a large encoder risks overfitting. Our model is applied to both the x-vector and wav2vec 2.0 embeddings, so the question becomes whether random slopes still help once the representation is stronger. If they stop helping, that says style has been absorbed into the representation, which is also informative.

Proposed Experiments

The primary benchmark is the UCLA Speaker Variability Database. It has read Harvard sentences and casual telephone conversation from the same speakers: the 40-female set used by Afshan et al., plus the expanded 50-male, 50-female set. We use five-fold speaker-disjoint cross-validation and report equal error rate on matched and mismatched trials.

The second benchmark is SRI-FRTIV, with four styles and three vocal-effort levels over two sessions. Here effort is crossed with style as a second slope factor. We test whether the model learns near-zero slopes for interview-versus-conversation, the pair Ferragne et al. found interchangeable, and large slopes for effort. That result would validate the model against a known finding.

A transfer check fits $\Gamma$ for read-versus-conversation on UCLA and applies it without refitting to SRI-FRTIV reading-versus-conversation trials. This asks whether the speaker-dependent style shift is a property of voices or of one corpus.

For the phonetic analysis, the Corpus of Spontaneous Japanese contrasts read, presentation, and dialogue speech. It lets us test whether speakers' embedding slopes track their own spectral-space compression, which is the quantity shown to predict recognition accuracy with correlation 0.97 in Nakamura et al.

The decisive checks are three. Does held-out speaker likelihood favor nonzero $\Gamma_k$? Does mismatched-trial error fall below fixed-offset PLDA with a bootstrap interval excluding zero? Do the estimated slopes correlate with independently measured acoustic reduction? A yes on the first two without the third would give an engineering gain with no phonetic explanation. A yes on the third alone would be a phonetics result with no verification payoff. All three together would support the core claim.