Learning Each Speaker's Prominence Cues from Unlabeled Speech via Cross-Cue Covariance

Listeners hear a word as prominent (stressed, accented, emphasized) because it stands out from its neighbours. It may be longer, louder, higher in pitch, or produced with a clearer voice. Speakers differ in which of these cues they use and how heavily. One speaker lengthens words, another raises pitch, and a third barely changes voice quality. A prominence detector trained on a handful of professional voices learns their particular mix of cues and treats it as universal. It then misreads new speakers, and non-native learners are the worst affected. We propose to estimate each speaker's cue mix at test time from a few minutes of that speaker's unlabeled speech. The estimate borrows a statistical trick from cosmology. Each acoustic cue is treated as a separate noisy "tracer" of a single hidden prominence signal. The speaker-specific strength of each cue is then read off the covariances between different cues. Noise that affects only one cue cancels in these cross-terms, much as instrument noise cancels when two independent sky maps are multiplied.

Put in two sentences: prominence is one hidden quantity that pitch, loudness, duration and voice quality each report with their own speaker-specific gain and their own private noise. Cross-cue covariances over unlabeled speech isolate those gains, which lets a detector reweight cues for every new speaker without labels. The question is whether this per-speaker calibration beats the implicit calibration that neural models already learn.

Core Claim

The analogy we borrow from cosmology is specific. A galaxy map clusters with amplitude $b^2\sigma_8^2$, where $b$ is the galaxy bias and $\sigma_8$ is the amplitude of matter fluctuations, so the two cannot be separated. Cross-correlating the galaxy map with a lensing map, which measures mass directly, gives $b\sigma_8^2$ and breaks that degeneracy. SDSS galaxy bias from halo mass-bias relation used galaxy–galaxy lensing to fix absolute bias for this purpose. DES Y3 × SPT/Planck cross-correlations showed that uncorrelated instrument systematics drop out of cross-spectra, and also that shared astrophysical foregrounds such as the thermal Sunyaev–Zel'dovich signal do not drop out and must be cleaned separately. Both lessons carry over directly.

Let $x_{s,c}(w)$ be the context-normalized value of cue $c$ for word $w$ spoken by speaker $s$. Context normalization here means the cue relative to neighbouring words, following the relative-salience literature. We model

$$x_{s,c}(w) = b_{s,c}\,P(w) + n_{s,c}(w),$$

where $P(w)$ is the latent prominence of the word, $b_{s,c}$ is how strongly speaker $s$ uses cue $c$, and $n_{s,c}$ is cue-private noise such as pitch-tracker error or intrinsic segment duration. When the noise is independent across cues, the covariance of two distinct cues is $C_{cc'} = b_{s,c}b_{s,c'}\sigma_P^2$. With three or more cues, the gains follow from off-diagonal terms alone:

$$b_{s,c}^2\sigma_P^2 = \frac{C_{cc'}\,C_{cd}}{C_{c'd}}.$$

The auto-variances $\mathrm{Var}(x_c)$ are then used only to read off each cue's private noise level $N_c = \mathrm{Var}(x_c) - b_{s,c}^2\sigma_P^2$. The speaker-calibrated prominence estimate weights cues in proportion to $b_{s,c}/N_c$. This is inverse-noise weighting, which is the standard optimal way to combine several tracers of one field. A second anchor plays the role of lensing. Prominence predicted from the transcript by a bidirectional language model, $T(w)$, is a tracer whose errors are linguistic rather than acoustic. Its covariance with each cue gives $b_{s,c}$ up to a constant shared by all speakers.

What would be new is the idea of a per-speaker production profile, meaning the vector of cue gains, estimated without prosodic labels and used at test time. Prior speech work either fixes the cue combination in training or leaves speaker calibration to a network's memory of a single utterance.

Relation to Prior Work

Three papers motivate the gap. Cue Interaction in the Perception of Prosodic Prominence regressed listener prominence ratings on duration, intensity, pitch and cepstral peak prominence (a voice-quality measure) in three German speakers. It found that one speaker did not use voice quality at all, and that dropping this speaker reshuffled the cue rankings. That is exactly a speaker-specific gain near zero, found after the fact with labels. Detection of Lexical Stress Errors in Non-Native English augmented training data with a single-voice synthesizer and attributed its limited robustness, an AUC of 0.62, to the lack of speaker diversity. Heard but Not Heeded showed that pretrained audio encoders entangle speaking style with content in ways a gradient-reversal layer cannot undo, and that downstream decoders fall back on text: Qwen2-Audio is 97.7% content-driven.

Two papers supply methods we reuse. The relative-salience literature supplies the context normalization that makes $x$ meaningful. Brenier et al. normalize within the intonational phrase. Bernhard et al. use neighbour-difference features normalized for speech rate and vowel type, reaching 96% accuracy on isolated nouns from six speakers. Quantifying the redundancy between prosody and text shows that relative prominence is predictable from bidirectional text context but leaves a residual text cannot explain. That is precisely the property a calibration anchor needs: it correlates with $P$, and its errors are independent of the speaker's acoustic habits.

Three papers are the closest competitors, and we differ from each in what is estimated and when. The role of context in neural pitch accent detection runs a bidirectional LSTM over the whole utterance so the network can adapt to a speaker's range, but the adaptation is implicit and confined to a single utterance. On the correlation between energy and pitch accent splits energy into 210 frequency bands and fuses them by majority vote with fixed weights, excluding pitch. Pitch Accent Detection improves Pretrained ASR drops explicit cues and fine-tunes wav2vec2 representations. None of the three estimates how a particular speaker distributes prominence across cues. Our proposal does not learn a new representation. It adds a small, interpretable calibration step that any of these detectors can consume.

Research Program

The first step builds the cue tracers. For each word we extract pitch, RMS intensity, duration and cepstral peak prominence, the last only where pitch tracking is voiced. All four are expressed relative to a symmetric window of neighbouring words, as in the relative-salience literature.

The second step is a cleaning step, analogous to removing thermal Sunyaev–Zel'dovich contamination. Some noise is not private to one cue. Vowel identity moves intrinsic pitch, intensity and duration together. Longer words and phrase-final lengthening shift duration and pitch jointly. Such shared structure would masquerade as prominence in the cross-covariances. We therefore regress out phone identity, syllable count, and position relative to phrase boundaries before computing covariances. We then check the result on words we are confident are unprominent, such as function words in repeated material, where the cross-covariances should be near zero.

The third step estimates the gains. For each speaker we estimate them from cue–cue cross-covariances, from cue–text cross-covariances, and from both jointly, with bootstrap uncertainty. We ask how many minutes of speech are needed before the estimates stabilize. The error of a covariance estimate shrinks roughly as $1/\sqrt{N}$ in the number of words, so we expect usable profiles within a few hundred words.

The fourth step feeds the gains to a detector in two ways. The first is a transparent route, the inverse-noise-weighted prominence score with a logistic threshold. The second appends the per-speaker gains and noise levels as conditioning inputs to the Nielsen-style detector and to a wav2vec2 detector.

A descriptive by-product comes from applying the estimator to hundreds of LibriTTS speakers, which carry no prosodic labels. This yields a population-scale picture of how much cue use varies across speakers, which to our knowledge nobody has measured without annotation.

Baselines

The naive reference point is the content-word rule of Nielsen et al.: label every content word as accented. It reports 82.2% accuracy on the Boston University Radio News Corpus (BURNC). We report it on every split, because any gain that fails to survive on the words this rule gets wrong is not acoustic.

The simplest strong acoustic baseline is per-speaker z-normalized cue concatenation. It uses the same cues and the same logistic or small-network classifier, but each cue is standardized per speaker instead of calibrated. This baseline removes offsets and scale but cannot learn that a speaker ignores a cue. It shares our features, classifier and tuning budget, so it isolates the calibration.

The strongest explicit-feature method is the CNN–BiLSTM of Nielsen et al., which reports 89.1% accuracy on BURNC with speech plus text. That split was not designed around held-out speakers. We will therefore retrain it under leave-one-speaker-out with the same hyperparameter search budget as our variants, and compare against that retrained figure rather than against 89.1%.

The strongest representation-learning method is the wav2vec2 detector of Pitch Accent Detection improves Pretrained ASR. It reports closing the F1 gap to the wav2TOBI baseline by 41%. We found no absolute F1 reported on Lacuna for this model, so we rerun it on our splits.

Four ablations isolate the key ingredient. The first uses population-average gains instead of per-speaker gains, which tests whether per-speaker estimation matters. The second uses principal components or diagonal-inclusive estimates instead of cross-only estimates, which tests whether excluding auto-variances matters. The third uses acoustic crosses only, with no text anchor, which tests the anchor. The fourth uses oracle gains fitted with held-out labels, which gives an upper bound.

We count the method as beating a baseline under three conditions: it improves leave-one-speaker-out F1 with bootstrap intervals that exclude zero, the improvement holds on the words where the content-word rule is wrong, and it concentrates on speakers whose estimated profile departs most from the training-pool average.

Why not just…?

Why not just z-normalize every cue per speaker? This is the real threat. If speakers differ mainly in offset and range rather than in which cues they use, z-scoring captures nearly all of the benefit. Evidence that it will not always suffice comes from the voice-quality-free speaker in Cue Interaction. For that speaker, z-scoring blows cepstral-peak noise up to unit variance and hands it the same weight as informative cues. Our z-normalized baseline and the population-gain ablation settle the question directly. If per-speaker gains add nothing over them on BURNC, the Boston Directions Corpus (BDC) and Bonn, the multi-tracer claim fails and we will report that.

Why not just let a full-utterance BiLSTM or a fine-tuned wav2vec2 learn the calibration implicitly? Implicit adaptation sees one utterance at a time. With four or five training speakers, it can learn only the cue mixes of those speakers. Heard but Not Heeded also finds that style and content stay entangled in pretrained encoders, which makes clean reweighting unlikely to emerge unprompted. Still, self-supervised models trained on thousands of speakers may already generalize. That is why we condition these same models on our gains and ask whether the conditioning helps. If it helps only the explicit-feature models, the contribution narrows to interpretability and low-resource settings.

Why not just use multi-speaker TTS augmentation, as Korzekwa et al. recommend? Augmentation spreads the training distribution but still asks the detector to recognize an unseen speaker's cue mix from a single word. It also depends on the synthesizer reproducing realistic cue trade-offs, which no paper in the file validates. We treat augmentation as complementary and use resynthesis only as a controlled test bed, not as a competitor we claim to beat.

Why not just predict prominence from text? The 82.2% content-word rule and the text residual documented by Wolf et al. show that text misses exactly the contrastive and given-information cases. In our design, text never labels a word. It only calibrates gains aggregated over hundreds of words, so a content-driven prior cannot override the acoustics of any single token. We verify this by reporting accuracy on accented function words and unaccented content words separately.

Proposed Experiments and Decisive Checks

The first check is recovery under controlled manipulation. We resynthesize BURNC utterances with pitch excursions scaled down, or with accent-related lengthening removed. This creates pseudo-speakers with known gain changes. The estimator must recover the direction and approximate size of each change, while z-normalization by construction cannot distinguish them.

The second check is the Bonn Prosodic Database. The estimator, run without labels, should assign a voice-quality gain indistinguishable from zero to the speaker whom Cue Interaction found not to use it. Across all three speakers, the estimated gain rankings should agree with the label-based regression rankings from that paper.

The third check is leave-one-speaker-out pitch-accent detection on BURNC (five speakers) and BDC (four speakers). The BDC transfer from read to spontaneous speech doubles as a test of whether cleaning holds up once disfluencies enter the signal. The BDC energy-voting system reports 81.9% accuracy on read speech.

The fourth check is a secondary test on stress-error detection, evaluated with AUC against the 0.62 of Korzekwa et al.. We use public L2-Arctic material where word-level stress can be annotated. Isolated words give short contexts, so we estimate each learner's profile from their full recording session. This tests whether calibration helps most where speaker variability is greatest.

The proposal fails cleanly under any of three outcomes: estimated gains are unstable across bootstrap resamples at realistic amounts of speech, cleaned cross-covariances on unprominent words stay far from zero, or per-speaker gains do no better than pooled gains.