Speaker Notes: Teaching Audio-Language Models to Retune Their Ears from Words

When a person hears an odd-sounding speaker say a word like "finance" with a fricative halfway between "f" and "s", the word itself tells them which sound was meant. After a handful of such words, they start hearing that speaker's ambiguous sound as "f" everywhere, including in places where no word can help, such as the minimal pair "fin" versus "sin". Psycholinguists call this lexically-guided perceptual learning: the vocabulary teaches the ear. Current audio-language models, meaning large language models that take speech as input, do the first half of this. Their language prior resolves an ambiguous sound inside a known word. They have no explicit way to do the second half, which is to carry what they learned forward to the same speaker's later speech without retraining.

We propose to give them a way: a short chain of thought (step-by-step text the model writes before answering) that acts as a gatekeeper and a ledger. For each utterance, the model reasons about whether the word context really disambiguates a doubtful sound. If it does, the model writes a speaker-specific note, for example "speaker B: ambiguous f/s is /s/, 6 consistent cases". On later utterances it consults those notes, and it can revise or retract them when the evidence turns. The claim is narrow and testable. Explicit, gated, speaker-scoped notes should reproduce the human recalibration effect on lexically uninformative test items. They should also do two things that weight-updating test-time adaptation cannot easily do: refuse to learn from misleading evidence, and keep opposite adjustments for two interleaved speakers apart.

Core Claim and What Is New

The human phenomenon is fast and specific. Interleaved lexical and audiovisual information can retune phoneme boundaries finds boundary shifts after about eight exposure items. Categorical perception of speech without stimulus repetition shows the lexical bias holds across 320 non-repeated morphed continua built with the STRAIGHT vocoder. We reuse that stimulus method directly. Computationally, Recognizing Speech in a Novel Accent: The Motor Theory of Speech Perception Reframed describes the listener as keeping a "conversion table" from a speaker's deviant sounds to native targets. Our proposal takes that metaphor literally and makes the table a piece of text in the model's context.

The closest machine work falls short in different ways. Visualizing Phoneme Category Adaptation in Deep Neural Networks reproduces retuning in a small feedforward network. After 40 ambiguous [l/ɹ] tokens, it labels the ambiguous token as the trained category 81.5% and 88.9% of the time. That result relies on supervised backpropagation with hard phone labels, and it has no speaker scoping. Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation applies chain of thought to speech. It finds that once the model writes its transcript, attention to audio collapses to near zero, a failure it calls "laziness", and the speech units themselves are frozen. Examining Test-Time Adaptation for Personalized Child Speech Recognition adapts a speech recognizer per speaker by minimizing prediction entropy. It documents that fillers and ambiguous audio "poison" the weights. CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training shows that intermediate evidence can regularize test-time adaptation, but only for image regions.

None of these combines three things: a top-down lexical check deciding which exposures count as evidence, a persistent per-speaker record of the inferred shift, and an evaluation on items where the lexicon cannot help. Without that last property, apparent "adaptation" is just the instantaneous word-context bias, which psycholinguists call the Ganong effect.

Research Program

The system has three parts.

The first is a frozen speech encoder. For each segment it exposes a short list of candidate phones with scores, and the language model can re-query the continuous features of any flagged segment. This follows the re-examination idea argued for vision in Position: Reasoning After Perception Means Reasoning Without Vision, and it addresses the attention collapse reported in Listening or Reading.

The second is a gate written as a brief chain of thought. For each word containing a low-confidence segment, the model lists the competing phones and checks which alternatives form real words. It then labels the item as disambiguating (exactly one alternative is a word), uninformative (both are words, or neither is), or suspect (the inferred shift conflicts with existing notes). Only disambiguating items update the ledger.

The third is the ledger. It is a compact, structured text block keyed by speaker and by phone contrast, holding a direction, a count, and a confidence. When the model decodes a doubtful segment, the active entries for that speaker shift its choice.

We train the gate and the note format with light LoRA fine-tuning on synthetic exposure episodes. LoRA trains small add-on weight matrices while the base model stays fixed. Following the finding in Chain-of-Thought Distillation with Fine-Grained Acoustic Cues for Speech Emotion Recognition that down-weighting reasoning tokens (1.0 on the final label, 0.01 on intermediate tokens) prevented an 8-point collapse, we weight losses on the decision tokens heavily and on the free-text reasoning lightly.

We also use the system as a scientific instrument. Human results, including Perceptual Learning in Lexical Tone: Phonetic Similarity vs. Phonological Categories and Phonological Symmetry Does Not Predict Generalization of Perceptual Adaptation to Vowels, show that retuning generalizes by acoustic similarity rather than by abstract phonological category. Because the ledger is discrete text, it naturally pushes toward category-level generalization. We will test whether that mismatches human behavior. If it does, we will try storing an acoustic exemplar reference in the note alongside the label.

Baselines

The naive reference is the same audio-language model decoding each test item with no exposure phase. Its behavior on the morphed continuum defines the unshifted boundary. Any recalibration claim must beat it on lexically uninformative items, not merely on words.

The first serious competitor is plain in-context exposure. All prior exposure utterances, audio and transcripts, are placed in context with no gate and no notes. This is the strongest "the model already does it" test, because a long-context audio model might adapt implicitly. It gets the same exposure items and the same context budget. We count it as beaten only if our system shows a reliably larger boundary shift on minimal-pair and isolated-syllable test items.

The second competitor is entropy-minimizing test-time adaptation (SUTA), as evaluated in Examining Test-Time Adaptation for Personalized Child Speech Recognition. That paper reports average word error rate (WER) on MyST dropping from 30.5% to 27.5% for wav2vec 2.0 base, and from 24.3% to 22.8% for the fine-tuned model. We run SUTA with its published single-utterance recipe on the same encoder, with the same number of exposure utterances. Beating it means matching or improving per-speaker WER on MyST while avoiding the degradation it suffers on contaminated exposure.

The third and strongest competitor is one we construct ourselves: lexically gated gradient adaptation. The same lexical gate selects disambiguating items, but the inferred labels drive a few gradient steps on small adapter weights in the encoder rather than being written as notes. This is the Scharenborg-style DNN recipe made unsupervised. It gets an identical gate, identical exposure, and a step and learning-rate budget tuned on held-out speakers. This baseline isolates the key ingredient, which is where the adaptation is stored. As a reference point for sample efficiency, the feedforward DNN reached 81.5% and 88.9% target categorization after 40 items, and humans shift after about eight.

Two ablations complete the set. One removes the gate, so every exposure updates the ledger. The other uses the ledger but replaces its content with random directions, which tests whether notes are actually read.

Why not just…?

Why not just run a recognizer and let a text LLM correct the transcript? A text-only corrector sees words, not the fact that one fricative was acoustically halfway. On a minimal pair like "fin/sin" it has no basis for choosing. Listening or Reading reports that cascaded and lazy chain-of-thought systems degrade sharply under 30% transcript corruption, while acoustically grounded ones stay stable. For a fair test we will still give the corrector the recognizer's n-best list and phone scores. We expect it to show the instantaneous word bias but zero carry-over to uninformative test items.

Why not just do gated test-time fine-tuning and skip the notes? This is a real threat. On a single speaker with clean exposure, the gated gradient baseline may well match our system, and if so the honest conclusion is that chain of thought earns its place as a gate, not as memory. We designed two conditions where the two storage methods should diverge. In the interleaved condition, two speakers with opposite shifts alternate every few utterances. Weight updates must fight each other, while speaker-keyed notes need not. In the retraction condition, early exposures are misleading and later ones reverse them. A note can be revised in one step, while weights carry momentum. If the gradient baseline wins these conditions too, we drop the storage claim.

Why not just use pretrained accent codebooks? Accented Speech Recognition With Accent-specific Codebooks reports up to 37% relative WER improvement on seen accents but up to 5% on unseen ones. CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice reaches only 68.5% accuracy separating similar Spanish accents. Our synthetic shifts are idiosyncratic to one invented speaker and belong to no accent category, so codebooks should not help by construction. The L2-ARCTIC test checks whether that conclusion holds for real speakers.

Proposed Experiments

The core experiment reproduces the human paradigm. Using STRAIGHT morphing as in the non-repetition study, we build English /f/–/s/ and Dutch /l/–/ɹ/ continua, the latter matching the Spoken Dutch Corpus setup of the DNN study. One group of exposure items puts the ambiguous token in /f/-biasing words and another in /s/-biasing words, with item counts of 4, 8, 20, and 40. Testing uses minimal pairs and isolated syllables along the continuum. The main metric is the difference in the proportion of /s/ responses between the two exposure groups, which is the human boundary-shift measure. We report it as a function of exposure count, alongside the human eight-item reference. No reported result was found on Lacuna for any audio-language model on this paradigm, so the no-exposure model is the anchor.

Robustness tests add contamination. We replace 30% of exposure items with nonwords, with contexts where both alternatives are real words, or with deliberately misleading items, mimicking the unreliable lexical access that harms non-native listeners in Phoneme category retuning in a non-native language. The gated system should keep its shift; the ungated system and SUTA should drift. We then run the interleaved two-speaker condition and the retraction condition. We also measure attention to audio during decoding with Value Zeroing, the attribution method from Listening or Reading, to confirm the model is still listening when it applies a note.

For natural speech, we use L2-ARCTIC: A Non-native English Speech Corpus. Its manual annotations of phone substitutions give real speaker-specific shifts. We expose the system to part of each speaker's utterances and measure WER and annotated-substitution recovery on held-out utterances from the same speaker. No reported WER was found on Lacuna for L2-ARCTIC; the codebook paper describes only zero-shot benefits without a number. On MyST we compare directly against the SUTA figures above.

The decisive check is causal editing. We flip the direction stored in a ledger entry while keeping the audio fixed. If the measured boundary shift does not flip with the edited note, the notes are decorative and the claim fails. Together with the interleaved-speaker comparison against gated gradient adaptation, this experiment decides whether explicit lexical reasoning is a real mechanism for perceptual learning or just a costly way to do what a few gradient steps already do.