Does Uncertainty-Carrying Symbolic Prosody Close the Gap for Audio Language Models?

Audio language models are large models that take speech in and produce text or actions. They often "hear" tone of voice in their internal layers and then ignore it when they answer. One study probed these models and found speaking style decodable at 82–85% accuracy from late encoder layers, yet final answers reached only 19–53%, and one model's tone predictions were 97.7% driven by the words rather than the sound (Heard but Not Heeded). An older idea from speech science offers a remedy: transcribe prosody (the melody, stress and phrasing of speech) into discrete symbols, such as ToBI labels. ToBI is a standard notation that marks which words carry a pitch accent, what shape that accent takes, and where phrases end with rising or falling tones. Those symbols can be written as text next to the transcript, where a language model cannot overlook them.

The difficulty is that the case for symbolic prosody rests mostly on hand-made labels, and it weakens once an automatic tagger produces them. Rosenberg found that manual ToBI sequences classify speaking style almost perfectly, while automatic AuToBI labels lost to a plain continuous pitch model on nativeness (Symbolic and direct sequential modeling of prosody). This proposal tests one specific claim: symbolic prosody helps downstream tasks only if the tagger's uncertainty is passed along with its labels. Instead of a single best guess per word, the model would see the tagger's top alternatives with their probabilities. We measure, on corpora that have gold human labels, exactly how much tagger error the symbolic route can absorb before continuous audio features become the better choice.

Core Claim

Three claims are to be validated empirically. First, a text-side prosody transcript built from automatic tagger posteriors will recover most of the benefit of gold ToBI labels on prosody-dependent tasks, while a hard 1-best transcript will not. Second, this soft transcript will beat continuous-feature and self-supervised discrete-token inputs in the regime where end-to-end models fail: little task data, unseen speakers, and inputs with the lexical context stripped away. Third, downstream accuracy falls off with label error along a measurable curve. We will map that curve by corrupting gold labels in controlled amounts, which turns the long-running "symbols versus signal" debate into a threshold that can be checked for any tagger.

The closest existing work is Hear2Act. It showed that asking an audio model to first write a free-text description of the user's inferred state raised the rate of optimal task outcomes from 15.3% (raw audio plus transcript) to 39.6%, against 40.7% with the ground-truth state. It tested only one binary cue, used only synthetic speech, and relied on the large model's own open-ended description. We differ on three points. The intermediate is a structured linguistic transcript produced by a small, separately trained tagger. It carries calibrated uncertainty. And the evaluation runs on human speech with gold prosodic labels, so we can tell errors caused by the representation apart from errors caused by the tagger.

Research Program

The tagger is Wav2ToBI (Wav2ToBI), trained on the Boston University Radio News Corpus. We extend it to output full posteriors over three things: whether each word is accented, which accent type it carries, and what break or boundary tone follows it. AuToBI serves as a second, weaker tagger. We calibrate each tagger's probabilities with temperature scaling on held-out data, so that a stated 0.6 means roughly 60% correct. We then render the output as inline text interleaved with the words. For a word whose accent is uncertain, a rendering might read NOT[H* .6 | L+H* .3], and a boundary might read %[L-L .8]. Only the top alternatives above a probability floor are shown, which keeps the sequences short.

The idea of "soft" matching comes from Unit selection and its relation to symbolic prosody. That paper argued that the mapping between prosodic meaning and sound is many-to-many: several acoustic forms can express the same intention, and one form can express several. It proposed fuzzy costs to reflect this but set them with ad hoc windowing functions and found no statistically significant effect. We take the concept and replace the hand-set functions with calibrated tagger posteriors, and we evaluate on understanding tasks rather than synthesis.

The core analysis is a label-noise sweep. On the Columbia Games Corpus and the Boston Directions Corpus, which both have manual ToBI labels, we corrupt the gold labels with the tagger's own measured confusion matrix, scaled from zero to double the measured error rate. Along the sweep we record downstream accuracy under hard and soft renderings. This lets us place real taggers on the curve. For reference, AuToBI on spontaneous dialogue reaches only 0.181 F1 on intermediate phrase boundaries and 35.34% accuracy on phrase-final tones (AuToBI). Wav2ToBI reaches 0.72 accent F1 and 0.79 boundary F1 on the Boston Directions Corpus when tested cross-corpus. With the curve in hand we can predict when each tagger's output is still worth using.

Baselines

The naive reference point is the transcript alone, with no audio. In Hear2Act this yields 14.6% optimal outcomes. On the other tasks we run it ourselves with the same language model and prompt budget. Any prosody representation that fails to beat it is carrying no usable information.

The strongest end-to-end competitor is continuous audio features passed into the language model. Concretely, WavLM-Large features go through a trained projector into the same model, following the stronger arm of Speech Discrete Tokens or Continuous Features?. That study reports 65.87% emotion-recognition accuracy for continuous WavLM with Llama3.1-8B, against 36.12% for discrete tokens. We also run Qwen2.5-Omni zero-shot on raw audio, which scored 53.7% on Expresso tone perception in Heard but Not Heeded. For fairness, the projector baseline and our tagger-plus-text system receive the same labeled task data, the same number of fine-tuning steps on the same backbone, and an equal hyperparameter search of 20 trials each. Beating this baseline means higher accuracy under the speaker-held-out and context-stripped conditions. We do not expect to beat it on the full in-domain training set, and we will report that condition honestly as well.

The strongest intermediate-representation competitor is Hear2Act's audio-inferred state: the audio model itself writes a free-text account of the user's prosody before acting. That setup reports 39.6% against a 40.7% ceiling. Because this leaves almost no headroom on Hear2Act itself, beating it there means matching roughly 39.6% with a far smaller tagger. The real contest is on human speech and on tasks with more than one prosodic dimension.

Self-supervised discrete tokens (k-means with 2,000 clusters plus byte-pair encoding, as in the comparative study above) test whether being discrete is enough, or whether the symbols need linguistic structure. Two ablations isolate our key ingredient. The first uses the same tagger with hard 1-best labels. The second uses gold ToBI, which gives the upper bound. The fraction of the gap between hard labels and gold labels that the soft rendering closes is the main quantity this proposal reports.

Why not just…?

Why not just let the audio model describe the prosody itself, as Hear2Act does? This is the most serious threat, because Hear2Act's two-stage pipeline already nearly reaches its ground-truth ceiling. The weakness is that free-text description depends on the same model that Heard but Not Heeded found to be content-driven. On Hear2Act's synthetic speech with a single binary cue, that dependence may not matter. We run the self-description arm on Expresso and on the gold-labeled corpora. If it matches our soft transcripts there, the structured tagger adds nothing beyond cost savings, and we will say so.

Why not just fine-tune the continuous adapter on the target task? With enough in-domain data, it probably wins. Continuous features beat discrete ones in five of six tasks in the comparative study. Our claim is limited to settings where end-to-end signals get lost: few labels, unseen speakers, and Expresso's four-speaker design, where content leakage dominates. The data-size sweep described below will show where the two curves cross.

Why not just use a better tagger's hard labels, or feed F0 statistics as text? Rosenberg's result shows that hard automatic labels can lose to direct pitch models. The hard-label ablation and a text-rendered pitch-statistics baseline (per-word pitch range and slope, given in numbers) test both options directly. If pitch statistics match soft ToBI, then the linguistic categories are not doing the work.

Proposed Experiments

On the Boston Directions and Columbia Games corpora, we replicate Rosenberg's classification of read versus spontaneous speech and of broadcast news versus dialogue. We compare gold, hard-automatic, soft-automatic, continuous, and discrete-token inputs at sequence lengths from 1 to 100 words, and we run the label-noise sweep on each. On Expresso, we measure style-recognition accuracy in two conditions: the original transcripts, and transcripts replaced with neutral text. The second condition tests whether soft transcripts reduce the reliance on text that makes Qwen2-Audio 97.7% content-driven. On Hear2Act, we run all arms on the released synthetic audio. We also re-record the feedback turns of 100 scenarios with human speakers to check whether conclusions drawn from synthetic speech hold for real voices.

Each arm is trained with 1%, 10%, and 100% of the task labels. We report accuracy with bootstrap confidence intervals and hold out speakers wherever the corpus allows. MSPB is Mandarin and would require a Mandarin prosody tagger, which the file does not provide. We therefore list it as a follow-up and do not claim results on it.

Three outcomes would settle the question. The first is whether soft rendering closes most of the hard-to-gold gap on the gold-labeled corpora, with a confidence interval that excludes zero. The second is whether the crossover point on the noise curve lies beyond Wav2ToBI's measured error rate, meaning today's taggers are already good enough. The third is whether soft transcripts beat the continuous adapter under speaker-held-out, low-data conditions. If soft labels close none of that gap, the symbolic advantage is an artifact of gold annotation, and the field should invest in making continuous signals reach the model's output instead.