Speech models are now trained on audio from hundreds of languages, yet nobody can say whether they register the pitch patterns that make an utterance a question rather than a statement, or that mark which word is in focus. Building labeled test sets for intonation takes expert phoneticians many hours per hour of audio, so such tests exist for a handful of languages at most. We propose a label-free probe that edits the pitch of ordinary recorded speech and leaves the words and timing untouched. It then measures whether a model's internal representations change more for edits that cross a linguistic category than for edits of the same acoustic size that do not. A category-crossing edit would be turning a final fall into a final rise. A non-crossing edit of the same size would be lowering the whole utterance's register. Because the probe needs only raw audio, it can be run on any language with recordings, including languages that have no writing system.
The organizing theory is the Autosegmental-Metrical (AM) model of intonation. It describes pitch as a sparse string of discrete High and Low targets, attached to stressed syllables ("pitch accents") and to phrase edges ("boundary tones"). How high those targets sit and exactly when they occur is gradient phonetic detail. That split between discrete targets and gradient detail is what the probe tests. A model that has learned a language's intonation should be more sensitive to changes in the targets than to equally large changes in the detail. A model that merely tracks acoustic pitch should not care which kind of change it sees.
Core Claim
We claim that a ratio of representation shifts can be computed without labels in any language, and that it predicts how well a speech model discriminates natural prosodic minimal pairs. The numerator is the shift caused by edits to the target tier. The denominator is the shift caused by phonetic edits matched in pitch displacement. The prediction should hold better than either raw pitch sensitivity or the amount of pretraining data per language. If this holds, the probe can extend intonation evaluation from the few languages with expert annotation conventions to the roughly hundred languages with public read speech.
The probe borrows its logic from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes. That paper swaps adjacent characters in raw text and measures how much a model notices. It reports a Pearson correlation of 0.99 with XNLI and Tatoeba performance across 44 languages, and it maps model blind spots across 350 languages. That probe works only on writing and only on linear character order. Ours moves the idea to audio and to the suprasegmental tier. It adds one ingredient the text probe lacks: a matched control perturbation that separates sensitivity to categories from sensitivity to any change at all.
What Exists and What Is Missing
Three lines of work motivate the gap. The Role of Prosody in Spoken Question Answering shows that an English spoken QA system barely uses prosody. It reports frame-level F1 of 36.0 on natural speech against 33.4 on pitch-flattened speech, and ~10.6 when a model trained on natural speech is tested on low-pass-filtered audio. That paper isolates prosody with a 300 Hz filter, which removes the syllable detail needed to locate pitch targets, and it covers English only. MMSU finds that the best speech LLM reaches only 53.6% on phonology perception, and that over half of all errors are perceptual. Its prosody tasks are English-centric, ad hoc, and multiple-choice. Fleurs-SLU covers 102 languages, but only for topic and comprehension tasks. On those tasks, cascades that transcribe speech to text and discard pitch remain competitive. So the multilingual evaluation that exists cannot see intonation, and the intonation evaluation that exists is not multilingual.
Two further lines constrain our design. The Zero Resource Speech Benchmark 2021 supplies the ABX discrimination method, a label-free test of whether a token is closer to a same-category exemplar than to a different-category one. We reuse it for our gold measure. Its spoken syntax pairs are flat synthetic speech, and that is the failure we avoid by editing real recordings. Text-Free Prosody-Aware Generative Spoken Language Modeling and ProsodyFM show that useful pitch representations must be relative and shape-based rather than absolute Hz. This is why our edits and our matching are done in semitones relative to each speaker.
Autosegmental Neural Nets is the closest use of autosegmental ideas in neural speech work. It shows that predicting lexical tone on a separate output tier helps tone recognition (Vietnamese tone error 33.91% versus 51.42%). It concerns lexical tone in four languages and recognition error rates, not sentence intonation or representation probing.
Phonological description supplies the categories we perturb. Malay phrase-level H and L tones are described in Intonational phonology and prosodic hierarchy in malay, Guarani tritonal accents in Paraguayan Guarani: Tritonal pitch accent and Accentual Phrase, and Korean phrase structure in A computational algorithm for F0 contour generation in Korean. Intonational features for identifying regional accents of Italian shows that peak timing, normally gradient detail, distinguishes Italian varieties. Its lesson is that the boundary between category and detail is partly language-specific, and the probe must not assume a universal split everywhere.
How the Probe Works
For each utterance, unsupervised signal processing finds syllable nuclei from intensity peaks within voiced stretches, and phrase edges from pauses longer than roughly 200 ms. This step needs no transcript. The pitch contour is then resynthesized with a standard source-filter vocoder in four versions. The first is a null pass with no change, so that vocoder artifacts are common to every comparison. The second is a tier edit, which changes the identity of a target at an anchor: final fall flipped to rise, an accent peak deleted, or an accent peak moved to the neighboring syllable. The third is a phonetic edit with the same total semitone displacement over the same frames, applied as a register shift or a scaling change that preserves target identity. The fourth is a random-jitter edit of the same displacement, with no reference to anchors.
Pitch edits leave duration unchanged, so model frames stay aligned one-to-one. Representation distance can therefore be computed frame by frame inside the edited region. For a model layer $r$, we define the categorical contrast score as
$$C = \frac{\mathbb{E}_u\, d\big(r(u_0), r(T(u))\big)}{\mathbb{E}_u\, d\big(r(u_0), r(P(u))\big)},$$
where $u_0$ is the null resynthesis of utterance $u$, $T$ is a tier edit, $P$ is the matched phonetic edit, $d$ is mean cosine distance over the edited frames, and the expectation runs over utterances in one language. A value of $C$ near 1 means the model treats category changes and gradient changes alike. Values well above 1 mean it separates them. We also report the numerator alone as raw sensitivity.
In lexical-tone languages such as Mandarin, Cantonese and Vietnamese, flipping a syllable's pitch can change the word. We therefore restrict tier edits there to phrase-final boundary regions and report these languages separately. This confound is itself informative, and we analyze it rather than hide it.
Research Program
The work has three parts. First, we build and validate the edit generator. Native listeners in about ten languages judge whether each tier edit changes the perceived category (question versus statement, focus position) and whether each phonetic edit does not. The perceptual dimension must be validated, because the probe is meaningless if humans do not hear tier edits as category changes.
Second, we collect a modest gold set of natural prosodic minimal pairs. These are recorded by multiple speakers per language, in languages with AM descriptions: English, German, French, Italian, Korean, Malay, Mandarin, Guarani, and two or three others where native-speaker phoneticians can be recruited. The contrasts are question versus statement, focus location, and phrase grouping. Model performance on these pairs is scored by cross-speaker ABX error.
Third, we run the label-free probe across the Fleurs audio underlying Fleurs-SLU for all of its roughly 100 languages and on several model families. These are self-supervised encoders (XLS-R, multilingual HuBERT), the Whisper encoder, the SeamlessM4Tv2 encoder, and an open speech LLM evaluated in MMSU (Qwen2.5-Omni-7B), probed layer by layer. The end product is a map of where each model hears intonational categories. We will also test whether the lexical dominance seen in English QA is a general pattern, which would show up as $C$ near 1 everywhere, or whether it tracks how much speech data a language has.
Baselines
The naive reference point is resource level alone: the hours of pretraining audio per language, or ASR error on Fleurs. If this predicts natural-pair ABX as well as $C$, the probe adds nothing beyond a data count. We fit both as predictors of gold ABX across model-language cells and compare held-out correlations.
The strongest existing competitor is a speech analog of the text local-structure probe from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes. In our setting this is the raw sensitivity numerator, or the random-jitter sensitivity, with no category-versus-detail control. That paper reports r = 0.99 against XNLI and Tatoeba for text. No reported result was found on Lacuna for any speech or intonation version. The two probes use the same utterances, the same vocoder, the same displacement budget and the same layers. Beating it means a significantly higher held-out correlation with gold ABX, tested by bootstrap over languages.
The key ablation removes metrical anchoring. Tier edits are applied at random syllables rather than at detected accent and phrase-edge positions, which isolates whether AM structure is what gives the probe its predictive power.
For speech LLMs we add an external reference: English MMSU phonology accuracy, where the best reported figure is 53.6% (MMSU). The check is whether English $C$ ranks models in the same order.
Why not just…?
Why not just perturb pitch randomly, as the text probe perturbs characters randomly? This is the real threat. Random-jitter sensitivity may already track how much a model attends to pitch, and if it predicts gold ABX as well as $C$, the AM framing is decoration. The direct comparison and the unanchored ablation settle it. We expect a gap because raw sensitivity rewards models that encode pitch as acoustics, and ProsodyFM's need to perturb absolute pitch suggests such encodings are common. This is an expectation, not a demonstrated result, and if the experiment fails we will report the simpler probe as the recommendation.
Why not just build a multilingual benchmark by synthesizing prosodic minimal pairs with text-to-speech? The sBLIMP pairs in The Zero Resource Speech Benchmark 2021 show what happens: flat synthetic prosody tests tolerance to synthesis artifacts. Most Fleurs languages also have no text-to-speech system with controllable intonation. Editing real recordings keeps natural voice and segmental detail, and the null-resynthesis control cancels vocoder effects. The human listening study measures whether the edits remain natural enough to be judged.
Why not just train a supervised probe to read pitch accents from model layers? It needs annotated targets, which exist only where annotation conventions exist. It also answers a weaker question. A linear probe can recover pitch from almost any acoustic representation, and recoverable information is not the same as information the representation is organized around; Chi et al.'s models plainly contain pitch yet ignore it. Our score asks whether the geometry of the representation already separates categories. Where English and German labels exist, we will run a supervised probe as a secondary comparison against gold ABX.
Why not just evaluate with ASR-plus-LLM cascades, which do well on Fleurs-SLU? Cascades discard pitch by construction, so they cannot be evaluated for intonation at all. Their competitiveness on topic tasks says only that current multilingual benchmarks do not require intonation.
Proposed Experiments and Decisive Checks
The first decisive check is perceptual validity. Listeners must categorize tier edits differently from the null version at clearly above-chance rates, and phonetic edits at near-null rates. Languages that fail this check are dropped from validation.
The second check is predictive validity. Across roughly ten gold languages and six models, $C$ must correlate with natural minimal-pair ABX error more strongly than random-jitter sensitivity, unanchored edits, and pretraining hours, with the advantage holding in leave-one-language-out evaluation. We will also test the Italian peak-timing family: the probe should show elevated sensitivity to alignment shifts in Italian, consistent with Savino's findings, but not in languages where timing is gradient.
The third check, run only if the first two pass, is the hundred-language map over Fleurs. Here we test two hypotheses. The first is that $C$ declines with resource level faster than ASR accuracy does, which would mean intonation is learned last. The second is that the instruction-tuned speech LLM has lower $C$ than its base encoder, mirroring the drop in grammatical agreement after instruction tuning reported by MultiBLiMP 1.0.
A null result on the second check, with $C$ no better than random jitter, would itself be useful. It would mean label-free intonation evaluation can rely on simple pitch sensitivity, and that AM structure buys nothing for diagnosis.
References
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- The Role of Prosody in Spoken Question Answering
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
- Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
- The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
- Text-Free Prosody-Aware Generative Spoken Language Modeling
- ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis
- Autosegmental Neural Nets: Should Phones and Tones be Synchronous or Asynchronous?
- Intonational phonology and prosodic hierarchy in malay
- Paraguayan Guarani: Tritonal pitch accent and Accentual Phrase
- A computational algorithm for F0 contour generation in Korean developed with prosodically labeled databases using k-toBI system
- Intonational features for identifying regional accents of Italian
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs