Minimal-Pair Negatives for Hearing Focus Through Tone

Mandarin speakers use pitch for two jobs at once. Pitch picks out which word is meant, since the tone of each syllable is part of the word. Pitch also carries what the speaker is doing with the sentence: which word is emphasized, whether it is a question, whether it is ironic. Today's speech-capable language models handle the second job badly. On the Mandarin Speech Prosody Benchmark, GPT-4o picks the emphasized word in only 40.00% of cases, while humans reach 92.15% (Can AI Understand Mandarin Speech Prosody?). Across the larger MMSU benchmark, more than half of model errors are failures to perceive the sound correctly, not failures of reasoning (MMSU).

We propose to teach an open audio-language model to separate the two layers in a fixed order. It first identifies the syllable tones. It then reads emphasis and sentence type as the deviation from what those tones alone would predict. Both steps run as two short sets of internal vectors rather than as written-out reasoning, so they add almost no response time. What keeps those silent steps tied to the audio is the choice of training contrasts. Each step is trained against negatives that are acoustic minimal pairs: recordings identical in every respect except the one layer that step is responsible for.

Core Claim

The claim is that factorized minimal-pair negatives are the missing ingredient. These are negatives that change only the lexical tones, or only the emphasis and boundary pattern. With them, a two-step "tone first, then deviation" decomposition can run in a compressed internal form without drifting back onto text priors. Random or generically "hard" negatives do not achieve this. The resulting model should close a substantial part of the focus and intonation gap on held-out benchmarks, at a response time close to answering directly.

The design follows how speakers themselves resolve the conflict. Emotional and sentence-type pitch is added as small perturbations at syllable edges on top of the lexical contour, rather than replacing it (Emotional Intonation in a Tone Language). Focus is carried jointly by lengthening, loudness and pitch range on the focused syllables, followed by compression on the syllables that come after them. The size of these effects depends on the underlying tone, including tone sandhi, the rule-driven tone changes between neighbouring syllables (Focus Acoustics in Mandarin Nominals). Conversational partners also shift their overall pitch range while preserving each word's internal tone shape (Absolute and relative entrainment in Mandarin conversations). That fact tells us which variations should count as "the same" for training purposes.

How the pieces relate to prior work

Three bodies of work supply the method, and each stops short of this setting. Hierarchical question decomposition shows that splitting a query into atomic sub-questions fixes many errors in frozen models. Is a Question Decomposition Unit All We Need? reports, for example, DROP F1 rising from 52.97 to 78.16 for GPT-3. Tree-structured scheduling of sub-questions in RoHT reaches 46.45 exact match on KQA Pro. Both operate on text, and they pay one generation pass per node. We keep only a fixed two-level tree, a tone node feeding a deviation node, because the phonology justifies that particular split. We do not learn open-ended decompositions; the same paper notes that automated decompositions are often illogical, which argues against learning them.

Latent reasoning methods remove the token cost of thinking. CoLaR compresses chains into continuous vectors and cuts reasoning length by 82.8% on MATH. MARCOS reports up to 15.7× speedup. SemCoT shows that silent vectors drift semantically unless contrastively anchored to reference reasoning text. All three compress linear text chains. SemCoT's anchor is text, which is exactly the wrong anchor for us: text is the source of the "context crutch" that MSPB documents and of the acoustic invariance that SpeechR documents. We anchor each latent node to audio contrasts instead.

Hard negative mining supplies the training signal. Easy negatives contribute vanishing gradient (Adversarial Contrastive Estimation). MoCHi synthesizes hard negatives by mixing embeddings. CoNT uses a model's own near-miss outputs as negatives. Embedding mixing has no notion of which acoustic layer changed, and it risks producing false negatives. The prosody contrastive models CLAPSpeech and CONCSS target speech synthesis with ordinary negatives. Our negatives are constructed in the signal itself, and they are factorized: what counts as a negative for one node counts as a positive for the other.

Research Program

We first build a minimal-pair corpus. Native speakers record sentence sets crossed by tone condition and information structure, following the design of Focus Acoustics in Mandarin Nominals: four tonal conditions including Tone 3 sandhi, crossed with focus types and noun-phrase positions. We extend the set to question versus statement and to boundary placement, and we record multiple speakers. We add pitch-resynthesized variants that move the focus deviation onto another syllable while keeping segments and lexical tones fixed. We also add tone-swapped variants that keep the focus and boundary pattern fixed. A native-listener panel labels a sample of every variant type. Variants that listeners do not perceive as intended are discarded, which directly addresses the false-negative risk that MoCHi-style mixing cannot check. Natural and resynthesized audio appear on both the positive and negative side of every contrast, so that resynthesis artifacts cannot serve as a shortcut.

The model is built on Qwen2.5-Omni-7B, which is open and has a published MMSU score. For each utterance it produces two small blocks of latent vectors before answering: a tone block $t$ and a deviation block $p$. During a first training stage, each block is decoded into a verbal sub-answer, a tone sequence and a focus/boundary label respectively, so that both are supervised in readable form. A CoLaR-style stage then compresses each block to a few vectors. Throughout training, a factorized contrastive loss holds both blocks to the audio:

$$\mathcal{L} = -\log \frac{e^{s(p,p^+)/\tau}}{e^{s(p,p^+)/\tau}+\sum_{p^-\in \mathcal{N}_{\text{dev}}} e^{s(p,p^-)/\tau}} \;-\; \log \frac{e^{s(t,t^+)/\tau}}{e^{s(t,t^+)/\tau}+\sum_{t^-\in \mathcal{N}_{\text{tone}}} e^{s(t,t^-)/\tau}}$$

The terms in the loss are defined as follows. $s$ is cosine similarity and $\tau$ is a temperature. For the deviation block, the positive $p^+$ comes from the same sentence with the same focus spoken by another speaker or with a shifted overall pitch range. The negatives $\mathcal{N}_{\text{dev}}$ are the focus-moved or boundary-changed variants with identical tones. For the tone block, the positive $t^+$ is the same tones under a different focus, and the negatives $\mathcal{N}_{\text{tone}}$ are the tone-swapped variants with identical focus. Each minimal pair therefore pulls one block together exactly where it pushes the other apart.

The analysis asks three things. We probe whether the deviation block encodes focus position independently of tone. We test whether moving focus in the audio changes the final answer, which is the direct counter-test to SpeechR's acoustic invariance. We measure where errors concentrate across tone conditions, with sandhi expected to be hardest given the variability documented in Tianjin Mandarin weak elements.

Baselines

The do-nothing reference is the backbone, Qwen2.5-Omni-7B, used zero-shot. It scores 60.57% overall on MMSU (MMSU). We found no reported MSPB result on Lacuna for this model, so we will measure it ourselves. The frontier reference points come from the cited papers: GPT-4o at 40.00% on MSPB focus marking and 44.38% on emotional prosody without context (MSPB), Gemini-1.5-Pro at 60.68% overall on MMSU with the best phonology-perception score of 53.6%, and Gemini-1.5-Pro at 67.68% on SpeechR. Beating the frontier models on MSPB focus marking and intonation while also improving on the backbone's MMSU phonology score would count as a clear result. Matching humans is not expected.

The strongest learned competitor is the same backbone fine-tuned on the same minimal-pair corpus with plain supervised labels and an equal number of gradient steps, with no decomposition and no contrastive term. A second learned competitor adds the contrastive term with in-batch negatives, in the style of CLAPSpeech. A third uses MoCHi-mixed embedding negatives. All three share data, compute and an equal hyperparameter-search budget. Our method must beat all three on held-out MSPB and the MMSU phonology tasks to support the claim that the type of negative matters.

The strongest engineered competitor is a cascade. Speech recognition supplies the words and tones. A pitch tracker supplies per-syllable pitch, which is normalized against a per-tone template estimated from neutral-focus recordings. The text backbone then receives these residuals as text. This cascade encodes the same phonological insight without any learning. A verbal two-node tree, written out in full, serves as an accuracy reference with high latency. CoLaR-style compression of a single undecomposed chain isolates the contribution of the tree. The key ablations remove one factor's negatives at a time, and swap the factorized negatives for random ones at equal count.

Why not just…?

Why not just fine-tune the speech model on prosody labels? This is the most likely way the proposal could turn out unnecessary, and it is the first learned baseline. Our prediction is that plain supervised training learns the frequent tone-focus combinations and fails on held-out combinations, especially sandhi. The reason is that nothing forces the tone and focus signals apart; a label loss gives no gradient for separating "this syllable rose because it is Tone 2" from "this syllable rose because it is focused." If supervised fine-tuning matches our method on held-out tone conditions, the hard-negative claim is false. We would then report the method as an efficiency result only.

Why not just extract pitch, normalize it by tone, and give the numbers to a text model? This alternative is a real threat, because it implements successive addition by hand. Its weaknesses are expected to come from three sources. Recognition and tone-labelling errors propagate downstream (Cao et al.). Focus is also carried by duration, intensity and post-focal compression, not by pitch alone (Focus Acoustics in Mandarin Nominals). Text models may not make good use of numeric contours. We will settle the comparison empirically, including a version of the cascade that is given ground-truth tones. If that oracle cascade wins, the honest conclusion is that perception should be engineered, and the latent design should be dropped.

Why not just prompt the model to reason step by step: first the tones, then the emphasis? Prompting reorganizes reasoning but cannot add perception the encoder lacks. MMSU attributes most errors to perception, and SpeechR shows outputs that do not respond to prosody changes. We will nevertheless run the prompt-only decomposition on the zero-shot backbone. If it recovers most of the gain, the training contribution is small.

Why not just use generic hard negatives such as embedding mixing? Mixed embeddings do not know which layer changed and may land on the anchor's own meaning. The MoCHi baseline at matched compute tests this directly, and the listener-validated negatives guard against false negatives in our own corpus.

Proposed Experiments

The decisive tests fall into four groups. The first measures accuracy on held-out MSPB tasks, with focus marking, intonation, prosodic disambiguation and emotional prosody without context as the primary targets, and the MMSU phonology and perception tasks as secondary targets. Because MSPB has only 178 items, it is used strictly for evaluation, and results are reported with bootstrap confidence intervals. Results are broken down by tone condition, including the tone conditions and sandhi contexts held out of training.

The second group is a counterfactual sensitivity test. Focus is moved by resynthesis in held-out recordings, and we measure how often the answer changes accordingly. The SpeechR acoustic-feature version provides a further check for acoustic invariance.

The third group covers latency: wall-clock time per query, number of generated tokens and number of latent vectors. We compare against direct answering, the verbal two-node tree and the single-chain latent model. The target is accuracy near the verbal tree at a cost near direct answering.

The fourth group checks transfer. MMSU reasoning tasks and SpeechR overall accuracy confirm that perception training has not damaged general reasoning.

The proposal is falsified in two cases. If factorized negatives do not beat in-batch and MoCHi negatives on held-out tone conditions, the core claim fails. If the oracle-tone cascade matches the method, the case for learning the decomposition fails.