Clock-Anchored Transport: Merging Separately Trained Audio and Visual Turn-Taking Models

When two people talk over video, the speech and the face are recorded on the same clock. Each moment of a conversation therefore gives a free, label-less pairing: what the audio model "thinks" at time t and what a separately trained vision model "thinks" at the same t. We propose to use that shared clock to merge two independently trained turn-taking models with different architectures, one listening and one watching, into a single predictor. Neither model needs to have seen the other's modality, and no turn-taking labels are needed for the merge itself. The motivating failure is concrete. Audio-only turn-taking predictors drop to near chance in noise. A jointly trained audio-visual model resists noise much better (Visual Cues Support Robust Turn-taking Prediction in Noise), but joint training needs large synchronized corpora that most new settings (in-car, robot, lab-recorded collaboration) do not have.

The core claim concerns how a merge behaves under small samples. Cross-architecture merging by optimal transport, meaning a soft matching between the neurons of two networks that conserves total matching weight, has so far matched neurons by feeding both models the same text (Transport and Merge). We replace "same input" with "same moment." We predict that the constrained, sparse transport map degrades gracefully when calibration data shrinks to the few minutes of synchronized video a small corpus can offer. An unconstrained learned mapping between the two models should overfit at that scale. If this crossover exists, it tells practitioners when post-hoc merging beats simply training a connector. If it does not exist, the experiment shows that clearly.

Core Claim

Let $a_i(t)$ be the activation of neuron $i$ in a chosen layer of an audio-only Voice Activity Projection (VAP) model, a model that predicts upcoming speech activity of each speaker from audio. Let $v_j(t)$ be the activation of neuron $j$ in a layer of a separately trained visual turn-taking model. The visual model reads OpenFace gaze, head pose and facial action units, and has a different width and depth. Over calibration moments $t$ drawn from synchronized but unlabeled conversation, we define a cost and solve an entropy-regularized transport problem:

$$C_{ij} = 1 - \mathrm{corr}_t\big(a_i(t), v_j(t)\big), \qquad T^\star = \arg\min_{T \in \Pi(\mu,\nu)} \langle T, C\rangle - \varepsilon H(T),$$

where $\Pi(\mu,\nu)$ is the set of matchings with fixed row and column totals, $H$ is entropy, and $\varepsilon$ controls how spread out the matching is. The merged model runs both encoders. At inference it adds $\lambda\, T^\star v(t)$ into the audio model's residual stream at the matched layer, and the layer pairing is chosen by the global layer-to-layer cost matrix as in Transport and Merge. Here $\lambda$ is a single scalar, set without labels by the generation-consistency criterion of AdaMMS applied to the predicted speech-activity distribution.

This is representation-level grafting, not coordinate-wise weight averaging. We say so plainly: with different input modalities, the donor's neurons cannot exist in the target without their input pathway. What is new is the source of correspondence. The time axis does the job that shared inputs do in Transport and Merge and that shared anchor samples do in From Charts to Atlas, which averages relative representations measured against common anchor inputs.

Positioning

Three lines of work motivate the gap without filling it. The Candor turn-taking study shows that visual cues buffer acoustic failure. It reports 84% versus 80% hold/shift accuracy clean, and 72% versus 61% at 10 dB music noise with noise-augmented training. It gets this only through a single model trained end to end, and it notes that generalization to unseen noise types is unpredictable. MULTICOLLAB combines sensor and speech models only by averaging output probabilities, reaching 67% on 176 binary instances. That corpus size is exactly where a jointly trained model cannot be fit. ELLSA keeps speech and action experts separate under shared attention, trained jointly.

Two lines of work supply the method. Transport and Merge provides the activation-based transport and layer pairing, and flags calibration sensitivity as its main failure mode; that failure mode is our central variable. AdaMMS provides label-free selection of the mixing coefficient. It also constrains us: it observed collapse when $\alpha > 0.7$, so we sweep $\lambda$ explicitly and report where predictions degrade.

Two lines of work constrain the design. An Empirical Study of Multimodal Model Merging shows that naive parameter merging of vision and language transformers fails without shared seed pre-training, which rules out weight averaging here. OptMerge argues that low-rank truncation removes noise from task vectors. Facial cues that predict a turn shift are subtle, so we test whether low-rank truncation of $T^\star$ discards exactly the directions that buffer noise. Multimodal Lego merges heterogeneous encoders without training through frequency-domain averaging at a single fusion point. It was tested on static biomedical tasks, has no neuron-level correspondence, and has no streaming setting.

Research Program

We train the two experts on the Candor corpus (850 hours of two-party video calls). The first is an audio-only VAP model with noise-augmented training, following the reported recipe. The second is a visual-only turn-taking model on OpenFace features, deliberately given a different hidden width and layer count. We use acoustic-only speech-activity targets throughout, which avoids the ASR-derived alignments that drove accuracy to chance in the original study.

We then compute transport plans on held-out, speaker-disjoint sessions under three factors. The first is calibration size, swept from a few hundred hours down to about ten minutes of conversation, roughly the scale of a MULTICOLLAB-sized lab corpus. The second is the acoustic condition the target sees during calibration: clean audio, so that the visual signal is matched to what clean-audio neurons should be doing, versus noisy audio. The third is synchronization: true alignment versus a deliberately shifted clock. For each plan we measure stability as the agreement of $T^\star$ across bootstrap resamples of calibration speakers. We then measure downstream hold/shift accuracy.

The generalization test is out-of-domain. We apply plans and experts trained on Candor to the Zoom audio and video of MULTICOLLAB's 24 instructor-builder pairs. There, turn-taking targets come free from voice activity and calibration data is genuinely scarce. No reported turn-taking result on MULTICOLLAB was found on Lacuna, so all numbers there are new.

Baselines

The naive reference is the audio-only VAP expert alone, which the Candor study reports at 80% hold/shift accuracy clean and 61% at 10 dB music noise with augmented training. Any merge that does not clearly beat this under noise has failed.

The strongest existing method is the jointly trained multimodal VAP model. Visual Cues Support Robust Turn-taking Prediction in Noise reports it at 84% clean and 72% at 10 dB music noise, with an effective gain of +10 dB SNR over audio-only. We retrain it on the same Candor training split as our experts, with matched parameter count and the same noise augmentation. On Candor it is an upper bound we do not expect to exceed. Success means recovering most of the gap between 61% and 72% at 10 dB using only unlabeled calibration data. In the small-calibration regime we also retrain it on the same ten minutes our merge receives; beating that version is the real target.

The most dangerous simple competitor is late fusion. We test fixed probability averaging, as in MULTICOLLAB, and a gated variant that learns one noise-dependent mixing weight from the same calibration minutes. We also run learned linear stitching: an unconstrained least-squares map from $v(t)$ to the clean-audio $a(t)$, fit on identical calibration data and injected at the same layer. This is the direct competitor to the transport constraint. MM-Lego's LegoMerge runs on the same two encoders with no extra training.

The ablations isolate the key ingredients. A clock-shifted calibration removes synchronization while keeping everything else fixed. A random plan with the same row and column totals controls for the extra branch alone. A rank sweep on $T^\star$ tests whether truncation removes buffering. Every method receives the same expert checkpoints, the same calibration sessions, and the same budget of label-free tuning for its single scalar.

Why not just…?

Why not just average the two models' predictions? This is a real threat. MULTICOLLAB's late fusion beat both of its components by a wide margin. Fixed averaging should fail in noise, because the audio model becomes confidently wrong and drags the average down. The gated variant can learn to discount audio under noise, though, and may match our merge. If gated late fusion equals the merged model on held-out noise types and on MULTICOLLAB, we will conclude that neuron-level merging adds nothing for turn-taking and report that result. The case for merging rests on intermediate injection letting visual evidence reshape the audio model's temporal state rather than only its final output. The test is whether the gap appears specifically at low SNR and with unseen noise types.

Why not just train a small connector between the encoders? The learned linear stitching baseline is exactly this, and with abundant calibration data we expect it to win, because it is less constrained. Our claim is a crossover: at minutes of calibration data, the transport constraint acts as a regularizer and the unconstrained map overfits to the calibration speakers. The calibration-size sweep, evaluated on speaker-disjoint and out-of-domain data, settles this directly. If no crossover appears, the proposal reduces to "use stitching," and we will say so.

Why not just train the multimodal model jointly? On Candor that is clearly better, and we use it as the ceiling. Most new settings lack 850 hours of synchronized video, and the generalizability literature in the file warns that small cohorts inflate within-cohort scores (Deception detection using a multimodal approach). The relevant comparison is therefore a joint model trained on the same ten minutes our merge sees, evaluated speaker-disjoint and on MULTICOLLAB.

Decisive Checks

Hold/shift accuracy is reported across test SNRs from −10 to +10 dB for babble, music and speech noise. One noise type is held out from all augmentation and calibration. We also report effective SNR gain relative to audio-only VAP. Four outcomes decide the proposal.

First, clock-shifted calibration must destroy the gain. If it does not, the transport plan is not using synchrony, and the benefit comes from the extra branch.

Second, the transport plan must beat learned stitching below some calibration size, while plan stability across speaker resamples stays high. Without this, the small-sample claim fails.

Third, the merge must beat gated late fusion on the held-out noise type and on MULTICOLLAB. Without this, merging is unnecessary.

Fourth, the rank sweep shows whether OptMerge-style truncation removes noise-buffering visual directions. We expect accuracy at low SNR to fall before clean accuracy does as rank shrinks. Such a split would be a concrete warning against applying task-vector denoising blindly to non-verbal channels.