The Agent's Voice as a Lever on Its Own Modality Gap

People talking to a voice assistant unconsciously drift toward the assistant's way of speaking. They copy its question intonation, its pacing, and sometimes its pronunciation of particular sounds. This drift is called phonetic accommodation. Separately, speech-language models, which are large language models that take audio directly as input, understand spoken input noticeably worse than the same content typed as text. Inside these models, internal representations of speech sit apart from the representations of the matching text, and this separation is called the modality gap. This note proposes connecting the two facts. The system that hears the user also chooses the voice the user hears. If users converge toward that voice, the agent's output voice partly decides what its next input will sound like. A well-chosen voice might pull user speech toward the region where the model's speech and text representations line up best.

The proposal is to test this directly in multi-turn spoken dialogue with real people. First, we measure how much of the speech-text gap inside an end-to-end model shrinks or grows as a user accommodates over a conversation. Second, we find out which accommodated features carry that change. Third, we check whether choosing the agent's voice to suit the listening model is a cheaper or more natural fix than the obvious alternatives: transcribing first, adapting the model, or converting the user's voice. The project matters because the gap is large. A recent analysis attributes an average 25% speech-versus-text performance drop in these models to two causes: 8.79% from eroded reasoning and 16.46% from poor speech-text alignment (Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models). That analysis also states that it covers only single-turn, synthetic speech.

Core Claim

Human accommodation toward an agent's voice is a partial, free form of input adaptation, and its effect on the model's internal speech-text alignment can be measured. The claim has three parts. First, within a single user, turns that are acoustically closer to the agent's voice will show better token-level speech-text alignment in the model and a smaller speech-versus-transcript accuracy gap. Second, this relationship is causal at the condition level. If the agent speaks with a voice that the listening model already processes with a small gap, users' later turns will be better aligned than if it speaks with a high-gap voice, and the difference will be mediated by measured acoustic convergence. Third, the benefit will come mainly from features that humans readily accommodate, such as intonation contour and speaking rate. It will not come from segmental features, which humans accommodate unreliably.

The novelty is the closed loop. Prior modality-gap work treats the input distribution as fixed and changes the model. Prior accommodation work measures human convergence but never asks what it does to the machine's internal representations.

Prior Work and Where This Sits

Several papers motivate the hypothesis because they establish that accommodation toward machines is real and feature-selective. In a Wizard-of-Oz study, a hidden human operator simulates a spoken dialogue system called Mirabella. Native German users increased rising-intonation wh-questions by 52% after the system used them, shifted 27% toward a dispreferred consonant variant, and showed no group-level shift on a vowel contrast (Phonetic Accommodation in a Wizard-of-Oz Experiment: Intonation and Segments). A follow-up with L1 French learners of German found a 33% consonant shift but no adoption of the German pitch-accent placement (Phonetic Accommodation of L2 German Speakers to the Virtual Language Learning Tutor Mirabella). In a shadowing study, expressive Alexa speech drew more perceived alignment than regular speech (Expressiveness Influences Human Vocal Alignment Toward voice-AI), and a related segmental shadowing study notes that synthesis artifacts can block convergence (Shadowing Synthesized Speech — Segmental Analysis of Phonetic Convergence). In meeting recordings, a speaker's estimated vocal-tract scaling factor depended on the previous speaker for 84% of speaker pairs (Applying vocal tract length normalization to meeting recordings). That finding shows partner-driven acoustic drift even at the level of spectral scaling. None of these studies connects the drift to what a recognizer or language model represents internally.

Other work constrains the design. When asked to clarify, speakers do not hyperarticulate to voice assistants. In the clarification study, the only significant change was vowel duration, at about 0.002 seconds (Speech Enhancement Patterns in Human-Robot Interaction: A Cross-Linguistic Perspective). This warns that some accommodation effects may be too small to matter to a model, so the program begins by estimating how large an acoustic shift must be before it matters. Gap analysis in contrastive models shows that fully closing the gap is not always optimal (Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning). We therefore measure task accuracy alongside alignment and do not treat alignment as the goal in itself.

The method comes from the speech-language-model analysis. Its Alignment Path Score (APS) traces each text token's best-matching speech positions inside the model. It then reports two measures along that path: cosine similarity, which captures direction, and Euclidean distance, which captures magnitude (Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models). That paper reports that APS correlates with the performance gap ($R^2=0.81$ for cosine and $0.72$ for Euclidean) and that speech and text align in direction but diverge in magnitude. We reuse APS unchanged on real conversational turns. We also ask whether accommodation affects the direction measure, the magnitude measure, or both. The speech translation work showing that the gap widens during autoregressive decoding (Understanding and Bridging the Modality Gap for Speech Translation) suggests tracking APS per turn across a dialogue rather than per utterance.

The closest benchmark-level work studies different things. SpokenWOZ (SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents) measures a task-level "spoken gap" in human-human dialogue and never measures representation geometry or convergence. VoiceBench (VoiceBench: Benchmarking LLM-Based Voice Assistants) applies programmatic perturbations to single utterances. AlignFormer (AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM) closes the length gap by compressing speech to text-token length, and in doing so it discards the prosody whose convergence we want to study. We use it as a contrasting model rather than as a remedy.

Research Program

The program has three stages, ordered so that a cheap failure early can stop the expensive human study.

Stage 1: a dose-response map built offline. For each model under study, we synthesize the text side of SpokenWOZ user turns with a pool of TTS voices and compute APS against the transcript. Candidate models are those analyzed in the modality-gap study, such as Qwen2-Audio and GLM-4-Voice, plus an AlignFormer-style compressed model if one can be reproduced. The voice with the highest APS is the model's "low-gap voice," and the voice with the lowest is its "high-gap voice." We then take real SpokenWOZ user audio and resynthesize it so that one dimension at a time moves a controlled fraction $\lambda \in [0,1]$ toward the low-gap voice. The dimensions are the F0 contour, speaking rate, vowel duration, and spectral scaling. We measure how APS and speech-versus-transcript accuracy change with $\lambda$.

Plotting the convergence magnitudes that humans actually produce on this curve tells us before any participant is recruited whether realistic accommodation is large enough to matter. The human convergence magnitudes come from the Mirabella and Alexa studies. If the curve is flat at those magnitudes, the closed-loop hypothesis fails cheaply and the note reports a negative result.

Stage 2: a closed-loop Wizard-of-Oz study. Participants complete SpokenWOZ-style task goals, such as booking a hotel or reserving a train, with a spoken agent. The agent's replies are chosen by a human wizard, as in the Mirabella design. Because a human picks the replies, dialogue flow does not depend on the model under test. The models analyze user audio offline. The design is within-subject and crosses two factors: agent voice (low-gap or high-gap for the target model) and prosodic style (regular or expressive, following the Cohn and Zellou manipulation). Each participant first reads a fixed set of task phrases as a pre-exposure baseline and reads them again after each block.

We plan about 60 participants, three times the Mirabella sample, because individual divergence was common there. All turns receive manual transcripts so that APS and transcript-input accuracy can be computed.

Stage 3: the analysis. Acoustic convergence is the change in distance between user and agent features from pre-exposure to in-dialogue turns, computed per feature and per turn. A perceptual AXB check validates it: listeners judge which of two user recordings sounds more like the agent. We fit mixed-effects models of per-turn APS (cosine and Euclidean separately) and of the accuracy gap. The predictors are convergence, turn index, voice, and style, with random effects for participant. A mediation analysis tests whether the voice effect passes through convergence.

Baselines

The naive reference point is the same model processing the same users' speech when the agent uses a neutral voice chosen without regard to the model. This is the high-gap or a mid-pool voice. We also use the pre-exposure readings, which carry no accommodation at all. Beating this reference means the low-gap voice condition yields a statistically reliable APS gain and a smaller speech-versus-transcript accuracy gap on the same task turns.

The strongest representation-level competitor is the inference-time angle projection from the modality-gap study. It rotates the least-aligned speech tokens toward their text counterparts and improved sd-qa accuracy by 7.52% for Llama3.1-8B and 4.19% for Qwen2.5-7B on the "Bottom3" tokens (Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models). It needs ground-truth transcripts, so we run it with our manual transcripts as an oracle upper reference on the identical turns. We do not expect voice steering to beat it outright. The informative questions are what fraction of its gain voice steering recovers without any transcript, and whether the two combine additively.

The strongest system-level competitor is a cascade of Whisper transcription and the same text LLM. VoiceBench finds such cascades often beat end-to-end models because transcription acts as an "implicit denoiser" (VoiceBench: Benchmarking LLM-Based Voice Assistants). On real spoken dialogue, SpokenWOZ reports zero-shot ChatGPT reaching only 13.75% joint goal accuracy, against 25.65% for the dual-modal SPACE+WavLM_aligned model (SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents). Those numbers are on human-human SpokenWOZ, not our human-agent corpus. No reported result exists for any system on human-agent accommodated dialogue, so these numbers set expectations only. We report the cascade's word error rate and accuracy under each voice condition, because accommodation may help the cascade too.

The model-adaptation competitor is LayerNorm-only tuning (Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning), which re-centers and re-scales representations and was reported to beat LoRA. We apply it on held-out participants' neutral-voice dialogue audio with matched data and a matched tuning budget. The ingredient-isolating ablation is voice conversion of user turns toward the low-gap voice before the model sees them. It delivers "full convergence" without involving the human, so it separates the effect of the target acoustics from the effect of the human doing the converging.

Why not just…?

Why not just use a cascade and skip the modality gap? This is a real threat for content accuracy, and we treat it as one. A cascade cannot use prosody. The addressee-detection study showed that prosody alone distinguishes machine-directed from human-directed speech at 0.886 unweighted recall (Speech and Text Analysis for Multimodal Addressee Detection in Human-Human-Computer Interaction). End-to-end models are worth fixing because they keep this information. Our experiments report cascade accuracy under every voice condition. If the cascade dominates end-to-end models on every condition, the practical recommendation becomes "use the cascade," and our contribution narrows to the scientific measurement.

Why not just adapt the model, for example with LayerNorm tuning, instead of steering the human? Model adaptation and voice steering act on different sides and could stack. Adaptation, however, needs in-domain data and a training run per deployment, and fine-tuning speech models erodes reasoning: the analysis attributes 8.79% of the speech drop to this. The head-to-head comparison and the combined condition in Stage 3 settle whether steering adds anything beyond adaptation.

Why not just voice-convert the user toward the model's preferred voice? This is the most serious alternative, and it is our key ablation. If conversion matches the low-gap voice condition, then the useful lever is a front-end, not the human loop. We would report that outcome. Two reasons it might fall short are testable. Conversion can strip the user's own expressive prosody, which we check with an emotion and addressee probe on converted versus raw turns. It can also introduce synthesis artifacts of the kind the shadowing literature identifies as disruptive.

Why not just simulate accommodation with perturbations? Stage 1 does exactly that, as a cheap filter. It cannot replace the human study, because real convergence is partial, varies across individuals, and differs by feature: intonation converges but vowels often do not. Uniform perturbations would misstate both the size and the profile of the effect.

Decisive Checks

Three results decide the claim. First, the Stage 1 curve must show a non-trivial APS change at convergence magnitudes in the range humans produce. Second, within participants, per-turn convergence must predict APS and accuracy-gap improvement after controlling for turn index, so that general fluency gains over a session are not mistaken for convergence effects. Third, the low-gap voice condition must outperform the high-gap condition on user-turn APS and accuracy, with a significant indirect path through measured convergence and with the effect concentrated in intonation and rate rather than segments.

A secondary check tests the Liang et al. warning. If the expressive voice increases convergence but lowers accuracy, raising alignment metrics is not the right objective, and voice choice should be tuned for task accuracy instead.