Phase-Locked Fusion of Speech and Transcript Under Onset Asynchrony

Speech systems that combine what was said (the transcript) with how it was said (pitch, loudness, timing) usually assume both streams arrive in step. Deployed systems break this assumption. A streaming speech recognizer emits each word some variable time after the audio that produced it, and when two people talk over each other, the system also has to work out which words go with which voice. Human perception handles this with a temporal binding window: a range of delays over which sound and a second signal are still treated as one event, and outside of which they separate. Psychophysicists measure that window by sweeping the stimulus onset asynchrony (SOA), the delay between the onsets of two stimuli, and fitting a curve to people's "same event" judgments. We propose running the same measurement on acoustic-linguistic fusion models. We also propose a fusion layer in which the binding window is an explicit, tunable dynamical quantity rather than something a network absorbs implicitly.

The core idea fits in two sentences. Give each audio frame and each transcript word a rotating phase, and let words pull their phases toward the audio frames they match, as coupled oscillators do. Words that lock are bound to that stretch of audio, and words that cannot lock within the coupling's reach are left unbound. Such a layer should tolerate a shared, drifting delay between speech and transcript while still refusing to glue a word to the wrong voice. The standard fix, training with randomly jittered delays, should buy tolerance only by blurring that precision.

Core Claim

Robustness to asynchrony has two sides: tolerance to lag, and precision in deciding what belongs together. Existing fusion evaluations measure neither. We claim that augmentation and learned attention trade one side against the other. We further claim that a Kuramoto-style phase-coupling layer (Kuramoto models describe oscillators that nudge each other's phases until they synchronize) achieves a better position on that trade-off. The layer's locking range acts as a controllable integration window, and failure to lock gives a built-in "these don't belong together" signal. The claim is falsifiable. If a cross-attention model trained with delay jitter matches the phase model on both lag tolerance and binding precision, the mechanism is unnecessary. The measurement protocol would then be the lasting contribution.

Where This Sits

Three lines of prior work motivate the proposal, and each leaves the same hole. The psychophysics supplies the measurement method. Audiovisual temporal sensitivity in typical and dyslexic adult readers swept audiovisual SOA from −440 to +440 ms in 40 ms steps and fitted Gaussians to simultaneity judgments. Two summaries came out: the point of subjective simultaneity, and the full width at half maximum (FWHM), which is the window width. The window was wider specifically for speech in dyslexic readers. When is indexical information about speech activated? used SOAs of −150, 0 and +150 ms to show that voice properties bind to words with a characteristic latency. Perceptual grouping of alternating word pairs and Judging temporal onset differences for concurrent vowels showed that pitch separation and onset timing jointly decide whether speech splits into separate streams. We borrow their stimulus logic and curve fitting, but apply them to models rather than people.

The oscillator literature supplies the mechanism. Artificial Kuramoto Oscillatory Neurons frames phase synchronization as a form of clustering, and Kuramoto Oscillatory Phase Encoding (KoPE) rotates Transformer queries and keys by evolving token phases. KoPE reports gains in convergence and multi-object segmentation, and names extension to other modalities as open. Tracking objects that change in appearance with phase synchrony and KomplexNet use phase as an identity tag that survives appearance change and overlap. All of these operate on images or puzzles. None handles two streams running on different clocks. Their known weaknesses constrain our design. KoPE depends on structured phase initialization, the CV-RNN needs a supervised synchrony loss, and KoPE costs about 20% more compute. For those reasons we initialize phases from real timestamps and keep the layer shallow. A Neural Oscillator Model of Auditory Selective Attention shows oscillators reproduce human streaming thresholds on synthetic tones, but it has no language side.

The fusion literature supplies the tasks and the baselines. Modulated Fusion using Transformer lets text keys and values drive acoustic attention (MAT) or modulate acoustic normalization (MNT). Learning Alignment for Multimodal Emotion Recognition from Speech learns soft word-to-frame attention and finds it beats hard forced alignment, because emotional prosody spills across word boundaries. MUSER concatenates encoder outputs. Easy Turn avoids the two-clock problem by having one model transcribe and then predict turn state. All of them are evaluated only at natural alignment.

Xu et al.'s result is the one that seems to argue against us. They found that soft, flexible alignment beats rigid alignment, which could suggest that more flexibility is always better. We read it differently: soft alignment wins because the binding window for prosody is wider than a word. That strengthens the case for measuring the window rather than assuming its shape.

Research Program

The first component is a model-psychophysics protocol. We start from IEMOCAP and from the turn-taking test set of Easy Turn. If that test set is unavailable, we use held-out MagicData-RAMC dialogues, the real-speech source of the Easy Turn corpus. We obtain word timestamps by forced alignment for clean transcripts, and from a streaming recognizer for realistic ones. From these we build three kinds of perturbation. A global SOA shifts every word timestamp relative to the audio by Δ, on the Francisco et al. grid of −440 to +440 ms in 40 ms steps. A drifting SOA lets the lag wander as a bounded random walk, which mimics variable recognizer latency. A local swap moves one word's timestamp onto another word's audio, or substitutes a transcript from a different speaker.

Models are then scored on three things: task accuracy as a function of Δ; a synchronized-or-not probe, which is the model analogue of a simultaneity judgment; and a word-attribution probe. In the attribution probe, the model must name which word carries the utterance's pitch-energy peak. The label comes from F0 and energy at forced-aligned boundaries. It is unchanged by a global shift but changed by a local swap. Fitting Gaussians to the probe outputs gives each model a point of subjective simultaneity and an FWHM, directly comparable to the human figures.

The second component is a two-talker binding test modeled on the streaming studies. We mix pairs of IEMOCAP utterances from the two speakers of a dyadic session. We vary the onset asynchrony between the talkers from 0 to 300 ms, and vary the pitch separation by resynthesis from 0 to 12 semitones. The model receives both transcripts and must predict each speaker's emotion, which is only possible if each transcript is bound to its own voice. This yields a binding surface over onset asynchrony and pitch separation. We can check it qualitatively against the direction of the human streaming effects: larger pitch separation and larger onset gaps should both make segregation easier.

The third component is the phase-locked fusion layer, inserted where MAT's cross-modal attention sits. Each audio frame $j$ carries a phase $\theta_j = \omega t_j$, where $t_j$ is its time and $\omega$ is a learned angular frequency near the syllable rate. Each word $w$ starts at $\phi_w = \omega \hat t_w$, where $\hat t_w$ is its arrival timestamp. The word then runs a few coupling steps:

$$ \phi_w \leftarrow \phi_w + \eta K \sum_j a_{wj}\,\sin(\theta_j - \phi_w), $$

Here $a_{wj}$ is the ordinary content-based attention weight between word $w$ and frame $j$, $K$ is a coupling strength, and $\eta$ is a step size. Attention logits are then multiplied by a rotary factor in $\cos(\theta_j - \phi_w)$, following KoPE's rotary mechanism. This lets content pull a mistimed word's phase back onto the matching audio, and the phase in turn sharpens attention. A phase-invariant readout feeds the classifier. The magnitude of each word's coupling term, $R_w = |\sum_j a_{wj} e^{i(\theta_j - \phi_w)}|$, indicates whether the word locked. A low $R_w$ marks the word as unbound, and we train it against the local-swap labels with a light auxiliary loss.

The layer makes a specific, testable prediction. The tolerated lag is limited by aliasing at half an oscillation period, so the model's FWHM should scale with $1/\omega$. Tolerance to drift should grow with $K$, as the classical locking condition of coupled oscillators predicts: a pair stays locked while their frequency mismatch is below the coupling strength. To remove the aliasing limit we add a slower second phase, as in The Gamma Slideshow's nested rhythms.

Baselines

The naive reference point is the same backbone (a pretrained speech encoder plus BERT) trained only on naturally aligned data with timestamps. We also report its text-only and audio-only versions, which show how much fusion matters at each Δ. Its lag-accuracy curve is the floor against which tolerance is judged.

The strongest published fusion method is MAT from Modulated Fusion using Transformer. It reports 0.74 unweighted F1 on IEMOCAP, above the 0.715 of its MulT baseline. It is the strongest competitor because it aligns by content with no temporal constraint, so it should be naturally tolerant of a global lag. It is also exactly the design that could fail at attribution and at overlap. We use the soft-alignment model of Learning Alignment as a second reference; it reports 70.4% weighted accuracy on IEMOCAP with 14.7%-word-error-rate recognizer transcripts. For turn-taking, the reference is Easy Turn, at 95.75% average accuracy, and 87.88% when the transcript branch is removed.

Every model we train, including our reimplementations of MAT and soft alignment, uses the same encoders, the same IEMOCAP session-wise splits, and the same hyperparameter search budget (an identical number of trials per method). Parameter counts are matched within 2% by widening the baselines. We report FLOPs, since the coupling steps add compute.

Two engineered baselines target the specific claim. The first is MAT trained with delay jitter drawn uniformly over the full ±440 ms test range, which is the most obvious fix. The second is a "lag estimator plus resync" pipeline: a small network, or a cross-correlation between the speech energy envelope and word timestamps, estimates the lag online, and the timestamps are shifted back before fusion.

Two ablations isolate the key ingredient. The first freezes $K = 0$, so phases are raw timestamps and the model is just timestamp-rotary attention. This isolates coupling from rotary encoding. The second replaces coupling with a single learned global offset, which isolates per-word dynamic locking from a constant correction.

Beating the baselines means three things together. At Δ = 0, the phase model must match MAT's clean score within noise, assessed over five seeds. On the tolerance-precision plane it must dominate the jitter-augmented MAT. That plane has area under the accuracy-versus-|Δ| curve on one axis, and attribution accuracy plus swap-detection AUC on the other. It must also beat the lag estimator under drifting lag.

Why not just…?

Why not just train cross-attention with random delay jitter? This is the real threat, and the experiment is built around it. Augmentation will almost certainly flatten the lag-accuracy curve. Our prediction is that it does so by widening the model's effective FWHM uniformly, which should show up as worse swap detection and worse two-talker binding at small onset asynchronies. We will train jitter at several ranges, trace its full trade-off curve, and ask whether the phase model lies above that curve. If it does not, we drop the mechanism claim.

Why not just estimate the lag and shift the transcript back? For a constant delay this is probably enough. The reason for building a model rather than using forced alignment is Xu et al.'s finding that hard alignment underperforms soft alignment for affect. A single global offset also cannot handle per-word drift, or two transcripts with different delays in an overlap. The constant-offset ablation and the drift condition settle this. If the estimator matches us under drift, the dynamic-locking claim fails.

Why not just use content-based attention without timestamps, as MAT does? Content matching is immune to a global lag by construction, so on utterance-level emotion under global shift MAT may be as good as anything. It has no notion of when, though. Our attribution probe and two-talker test are cases where content cues are weak (recognizer errors, similar words from two voices) and timing must decide. If MAT solves them, then timing is not needed for binding in speech-text fusion. That would be a useful negative result.

Why not just generate the transcript inside one audio model, as Easy Turn does? Single-model transcription removes the second clock, and Easy Turn's 95.75% shows how well that can work. It does not cover the common deployment case, where a separate recognizer or an external transcript feeds a lightweight fusion model, and it does not cover multi-talker overlap. We will report where the protocol applies. We can also run the SOA sweep on an Easy Turn-style model by delaying audio relative to its own decoding context, to check whether it has an implicit window.

Decisive Checks

Four outcomes decide the proposal. First, the phase model either dominates the full jitter-augmentation trade-off curve on IEMOCAP or it does not. Second, its fitted FWHM either scales with the oscillation period when $\omega$ is varied, or the phase layer is not the thing setting the window. Third, under drifting lag the $K = 0$ and constant-offset ablations should fall behind the coupled model. If they do not, coupling adds nothing beyond rotary timestamps. Fourth, on the two-talker test, binding accuracy should rise with pitch separation and inter-talker onset gap in the same direction as the human streaming results. If the first check fails, the contribution narrows to the SOA-sweep protocol and the tolerance-precision measurement, which remain new: no model in this file has had its window of integration measured.