Text Arrives First: Measuring and Reordering How Audio-Language Models Acquire Prosody

Speech carries two signals at once: the words, and how they are said (pitch, loudness, timing, tone). Audio-language models, which are large text models fitted with an audio front end, keep the first and lose much of the second. Their audio encoders still hold tone of voice: linear probes on late encoder layers decode it at 82–85% accuracy. Yet the generated answers ignore it, landing at 19–53% (Heard but Not Heeded). When the transcript contradicts the voice, models follow the words, and accuracy can even rise when the audio is played backwards so the words become unintelligible (VoxParadox). This proposal treats that failure as a problem of order of arrival. During training, the solution that reads the words is found long before the solution that listens to the voice. Once it is found, it occupies the decision and blocks the acoustic solution from forming, even when the acoustic solution would be more accurate.

The idea comes from evolutionary biology. There, "arrival of the frequent" means that outcomes produced by many different genotypes are discovered first, in time roughly inverse to their frequency. Populations then lock onto them and stop exploring better but rarer alternatives (The Arrival of the Frequent). If the same dynamics govern how audio-language models learn prosody, two consequences follow, and both can be tested. First, the training step at which a model acquires a vocal cue should be predictable from how rare that cue's use is among randomly initialized or randomly perturbed networks. Second, the order of exposure should matter in a way that plain data mixing cannot reproduce. A model that finds the acoustic solution before the text solution is available should keep using it afterwards. We measure audio reliance as a smooth dose-response curve: how strongly the model follows the voice as the vocal cue grows from subtle to exaggerated. We fit these curves with restricted cubic splines, a curve-fitting method borrowed from epidemiology that is good at locating thresholds.

Core Claim

Paralinguistic loss in audio-language models is a lock-in caused by arrival order, not a lack of encoded information. Under equal data and compute, an "acoustic-first" schedule should leave a lasting, larger gain in audio reliance than mixing the same examples together. The schedule first trains on audio whose words are scrambled, then adds intelligible speech. The gain should be measurable as a shift of the dose-response threshold, the smallest vocal deviation at which the model trusts the voice over the transcript, toward smaller deviations. The ordering-versus-mixing comparison is the decisive one. If mixing matches ordering, the lock-in account is wrong, and the simpler data-balance explanation wins.

Where the Pieces Come From

Three lines of work motivate the claim. The evolutionary literature supplies the prediction that discovery time scales as the inverse of frequency, $T(p) \propto 1/F(p)$, where $F(p)$ is the fraction of genotypes producing outcome $p$. It also supplies the observation that frequent outcomes can fix in a population even when fitter ones are reachable (The Arrival of the Frequent). It ties frequency to simplicity: outcome probability decays exponentially with descriptive complexity (Symmetry and simplicity spontaneously emerge). All of this was shown on RNA folds, protein complexes and similar systems, never on learning machines.

The deep-learning literature makes the same volume argument for networks, without the time-of-arrival dynamics. Functions that many parameter settings produce are simple and dominate the prior over what a network computes (Deep neural networks have an inbuilt Occam's razor). When a simple and a complex feature are both predictive, networks latch onto the simple one and become insensitive to the other, even on balanced data (The Pitfalls of Simplicity Bias). A density-of-states study constrains how far we can push this. Parameter volume alone explains generalization only partly, and the gap to gradient descent is roughly 30% at 30 training samples and about 12–13% at 600 (Revisiting the Volume Hypothesis). So our volume-based predictions of acquisition time may hold only coarsely, and we treat that as a hypothesis to test, not an assumption.

The speech literature supplies the phenomenon and the benchmarks, but it frames the failure as a static deficit, not a training-time race. VoxParadox and Heard but Not Heeded both document the gap between what the encoder holds and what the model outputs. Heard but Not Heeded finds Qwen2-Audio 97.7% content-driven. VoxParadox repairs the failure after the fact with a learned mixer over encoder layers plus preference tuning. The two papers also disagree about where vocal information peaks: VoxParadox places it in middle layers 5–15, Heard but Not Heeded in the top third of the encoder. Our layer probes across training checkpoints will report which pattern our models show, but resolving that dispute is not the goal.

Restricted cubic splines supply the measurement method. These are piecewise cubic curves joined smoothly at chosen points called knots, and forced to be straight lines beyond the outer knots so they do not swing wildly where data are sparse (Restricted Cubic Splines for Non-Linear Risk Modeling). Clinical studies use them to find thresholds, such as a 25 g/day sugar intake above which obesity risk accelerates (Sugar Is the Key Cause of Overweight/Obesity in SSB). They also find U-shaped risks, such as depression rising with both short and long sleep (night sleep duration and depression, CHARLS 2020). We use splines only to measure, never as model inputs. We do not expect a U-shape in advance; it is one possible outcome. If very exaggerated cues push audio out of the training distribution, reliance could fall again at the extreme. A standard S-shaped curve would hide that, and a spline would show it.

Research Program

The work has three stages that share one stimulus set. We take Expresso, a 47-hour expressive speech corpus with 4 speakers and 26 styles (Expresso), and build cue-conflict items in the style of VoxParadox. In each item the transcript asserts one attribute and the audio carries another. The vocal cue is manipulated by signal processing along a continuous "dose." We use two kinds of cue. Level cues are global and simple: mean pitch shifted in semitones, loudness shifted in decibels. Contour cues depend on timing and shape: the direction of the final pitch rise or fall, and where stress falls. Contour cues include prosody-only distinctions like the Korean wh-word case, where the same text means "where" or "somewhere" depending on pitch (Prosody in Cascade and Direct Speech-to-Text Translation). The arrival account predicts that level cues are acquired early and contour cues late or not at all.

The first stage is a controlled test of whether frequency predicts arrival time. It uses small audio-text models trained from scratch, with text and acoustic cues equally predictive and data balanced. For each cue we estimate its frequency $F(c)$ as the fraction of random initializations, or random small weight perturbations around a pretrained checkpoint, whose outputs change measurably when only that cue is altered. We then record the step $T(c)$ at which the trained model's audio reliance on that cue crosses half its final value, and test the transplanted prediction

$$\log T(c) = -\alpha \log F(c) + \beta,$$

where $\alpha$ is the fitted slope. Arrival of the frequent predicts $\alpha$ near 1, because $T \propto 1/F$. Following the Level III failures reported for complex biological maps (Bounding phenotype transition probabilities via conditional complexity), we expect the ordering of cues to be correct before the slope is.

The second stage measures dose-response in pretrained models. For each model and checkpoint, audio reliance is the probability $R(d)$ of answering according to the audio rather than the transcript at cue dose $d$. We fit $\mathrm{logit}\,R(d) = f(d)$ with a restricted cubic spline at four knots placed at the 5th, 35th, 65th and 95th percentiles of dose. From the fitted curve we report two quantities with bootstrap confidence intervals: the detection threshold $d^\ast$, the smallest dose where reliance exceeds 50%, and the plateau level. We do this for Qwen2-Audio and Audio Flamingo 3 before and after each training intervention.

The third stage tests whether reordering beats mixing. We fine-tune Qwen2-Audio with low-rank adapters, small trainable weight additions, under four schedules that use identical examples and steps and differ only in order. The standard schedule uses intelligible speech throughout. The mixed schedule randomly blanks or scrambles content in half the batches. The acoustic-first schedule trains on content-scrambled audio first, using reversal and syllable shuffling, then switches to intelligible speech. The text-first schedule is the reverse of acoustic-first, and lock-in predicts it should be worst. The adapter placement leaves the encoder trainable, which addresses the concern that frozen-encoder patches are capped by whatever the encoder retained.

Baselines

The do-nothing reference is the untouched base model. On VoxParadox, Audio Flamingo 3 scores 17.40% and Qwen2-Audio 14.85% (VoxParadox). On the MMSU paralinguistic subset, Audio Flamingo 3 scores 37.74%. On full MMSU, the best open model, Qwen2.5-Omni-7B, reaches 60.57% against 89.72% for humans (MMSU). Every intervention must clear these numbers on the same splits.

The strongest existing method is VoxParadox's prompt-conditioned layer mixer combined with preference tuning. It reaches 72.30% on VoxParadox for Qwen2-Audio and 65.20% for Audio Flamingo 3, and 54.78% on the MMSU paralinguistic subset for Audio Flamingo 3 (VoxParadox). It is the strongest competitor because it targets exactly the transcript-over-voice conflict we study, on the same models. We will rerun it with the published recipe on our Expresso-derived training data. It gets the same example count and an equal hyperparameter search budget, eight configurations per method chosen on a shared validation split. Beating it means the acoustic-first schedule exceeds it on VoxParadox and the MMSU paralinguistic subset by more than the spread across three seeds. It also means a lower detection threshold $d^\ast$ on held-out contour cues, with speech-recognition word error rate on Expresso within 0.5 points of the base model. We also report the two methods combined. Arrival ordering and post-hoc mixing may be complementary, and that is a useful result either way.

The ablation that isolates the key ingredient is the mixed schedule. It sees exactly the content-scrambled examples the acoustic-first schedule sees, only interleaved. The text-first schedule is the second ablation. The gap between acoustic-first and text-first, at matched data and compute, is our direct measure of lock-in. A data-volume baseline that simply adds more paralinguistic examples, at double the steps, shows how much of any gain plain data can buy.

Why not just…?

Why not just call it data imbalance and add paralinguistic training data? This is the most serious alternative. Text-pretrained backbones have seen vastly more words than voices, and that alone could explain the preference. Simplicity bias does persist on balanced data (The Pitfalls of Simplicity Bias), and that is evidence against data imbalance being the whole story, but it comes from vision and synthetic tasks. The proposal settles the question with three experiments. Stage one balances data from scratch. The double-data baseline tests whether volume closes the gap. Most decisively, imbalance alone predicts no difference between acoustic-first, mixed and text-first schedules that share identical examples. If the three tie, we report that arrival order does not matter here.

Why not just randomly mask or scramble transcripts during training? This is exactly our mixed schedule, so it is both a baseline and the critical ablation. If it matches acoustic-first, the useful finding is "use content scrambling," and the lock-in story adds nothing. We say so in advance.

Why not just use the layer mixer and preference tuning from VoxParadox? It works well on VoxParadox, but it leaves a 54.78% ceiling on the MMSU paralinguistic subset. MMSU also attributes over half of model errors to failures of perception, which a decision-level patch on a frozen encoder may not reach. Our comparison and the combined run show whether ordering adds anything beyond it. On VoxParadox's own synthetic items it may not, so we weight held-out contour cues and MMSU more heavily.

Why not just fit a logistic psychometric curve instead of splines? A logistic curve is the standard choice and may suffice. We fit both and compare them by AIC, an information criterion that penalizes extra parameters. If the logistic fits as well, we report thresholds from the logistic. The spline matters only if reliance turns down at extreme doses, which an S-shaped curve cannot represent.

Proposed Experiments and Decisive Checks

Three results would confirm the account. First, the ordered gap in VoxParadox accuracy and in $d^\ast$, from acoustic-first through mixed to text-first, exceeds seed variance on Qwen2-Audio, while word error rate stays flat. Second, in stage one, frequency correctly ranks when cues are acquired across at least five cue types, with level cues before contour cues, and the slope $\alpha$ is estimated with a confidence interval. Third, across checkpoints, the threshold falls once and stays down after intelligible text is introduced in the acoustic-first run, instead of drifting back.

Two results would refute it. If the three schedules tie on matched examples, the lock-in mechanism is not operating. If frequency fails to rank arrival times even in small from-scratch models, the evolutionary analogy carries no predictive weight for learning. Either outcome is publishable as a constraint on how the speech community should frame the transcript shortcut.