Spoken dialogue systems have to decide, many times a second, whether the person they are talking to has finished speaking. The best current models do not wait for silence. They predict ahead. Voice Activity Projection is the main example: a self-supervised network that listens to both speakers' audio channels and forecasts who will be speaking over the next two seconds. These models are trained mostly on decades-old telephone corpora, and they break when the conversation changes character. On the multi-domain TurnBench benchmark, false interruptions rise sharply in casual talk full of "uh-huh"s. The obvious fix comes from source-free adaptation, which updates a deployed model on unlabeled target data without access to its training set. That fix has a known weakness. Its standard objective, making predictions more confident, is exactly wrong for turn-taking, because the onset of a backchannel and the onset of an interruption are genuinely ambiguous.
The observation behind this proposal is that turn-taking needs no confidence proxy. The quantity Voice Activity Projection predicts, who speaks in the next two seconds, is simply observed two seconds later. Every conversation therefore produces its own exact training labels, for free and with a fixed delay. We propose to adapt a frozen-architecture turn-taking model online, during a conversation, by minimizing its error on these delayed self-labels. We update only a small set of normalization parameters, which are anchored to the source weights. Because the model adapts on the very stream it is scored on, we treat the evaluation the way contamination research treats leaked test data. Every prediction is scored before its label is ever used for an update. Controls with mismatched futures separate genuine adaptation to conversational dynamics from within-conversation memorization.
Core Claim
Delayed future voice activity is a source-free adaptation signal that is unbiased with respect to the model's own beliefs. We claim it will cut interruption false positives on conversational styles unlike telephone speech, and that entropy minimization cannot do the same, because it amplifies whatever the model already believes about ambiguous overlaps. We also claim the gain goes beyond recalibrating a decision threshold. Adaptation should change when the model's probability of a turn shift rises, not only how high it must rise before the system acts. If a scalar threshold tuned online on the same delayed labels matches the full method, the claim fails, and we will report that calibration is sufficient.
Prior Work and Where This Sits
The turn-taking literature motivates the problem. Schlangen frames the core tension: systems that react to silence are either too slow or interrupt too often, so they must predict instead. Voice Activity Projection answers this with a stereo-audio transformer. It discretizes the next two seconds into eight speaker-by-time bins, giving 256 joint states, and learns from untranscribed audio. This property is what makes our method possible: its training target can be read directly off future audio. TurnBench supplies the evidence for the gap. It covers 30 hours across six styles (casual, task-oriented, instructional, collaborative, argumentative, narrative). It shows interruption false positives varying widely by style, and it shows humans starting their turns a median 151 ms before the other speaker stops. TurnBench is a static benchmark and proposes no adaptation. Talking Turns documents the "backchannel black hole" in audio foundation models such as Moshi and GPT-4o cascades. It builds a supervised offline judge, which has no mechanism for changing an agent's behavior during deployment. Deployed duplex systems such as Freeze-Omni, Proact-VL, and the text-only RC-TurnGPT all fix their turn-decision heads after offline training. We add the missing online component and leave their architectures untouched.
The source-free adaptation literature supplies the method and also constrains it. Tent updates only normalization scale and shift parameters to minimize prediction entropy on test batches. UBNA re-estimates normalization statistics. LCCS confines the update to a low-dimensional subspace to keep it stable. We borrow their choice of what to update: a small, normalization-sized parameter set. We reject their objective, for two reasons. Entropy minimization depends on confident predictions being correct, which fails on ambiguous overlaps. Batch statistics also need independent batches, which a single streaming conversation cannot provide. Kuzborskij et al. give the theoretical licence for adapting from a source hypothesis without source data. Their stability argument motivates our explicit anchor to the source weights. Input-space adaptation for frozen predictors offers an alternative for certified models that cannot be modified, and we include it as a variant. Test-time training for ventilation prediction is the closest analogue in spirit: adaptation on physiological time series across hospital sites. It does not exploit labels that arrive with a fixed delay.
The contamination literature supplies the evaluation discipline. DyVal showed that models with strong static-benchmark scores can drop to near zero on freshly generated instances, so a score earned on familiar data says little. We apply the same suspicion in two places. First, a model adapting on the test stream can trivially "learn the answers," so our scoring is strictly predict-then-update. Second, Switchboard and Fisher have circulated for decades. We therefore compare frozen-model performance on these corpora against TurnBench's newly recorded audio, under acoustic disguises that preserve timing, as a modest probe of memorized acoustics.
Research Program
The objective is a delayed log-likelihood. At time $t$ the model with parameters $\theta$ sees audio $x_{\le t}$ and outputs a distribution over the 256 future states. At time $t+\Delta$, with $\Delta = 2$ s, the realized state $s_t$ is read deterministically from voice activity on both channels. We minimize
$$ \mathcal{L}(\phi) = -\frac{1}{|B|}\sum_{t \in B} \log p_{\theta_0,\phi}(s_t \mid x_{\le t}) + \lambda \lVert \phi - \phi_0 \rVert^2 , $$
The terms in this objective are as follows. $\phi$ are the layer-normalization scale and shift parameters, initialized at their source values $\phi_0$. $\theta_0$ are all other weights, which stay frozen. $B$ is a small replay buffer of recent time steps whose futures have been revealed. $\lambda$ controls how far adaptation may move from the source model. Layer normalization uses per-frame statistics, so the method needs no batches. Updates run every few seconds on the buffer.
We will build three variants that share this signal. The first is per-conversation adaptation that resets at the end of each dialogue. The second is persistent adaptation that carries $\phi$ across dialogues of the same style. The third is an input-space version that leaves the network fully frozen and learns a light front-end transform of the audio features.
Live deployment adds a complication. Once the system itself speaks, the observed future partly reflects its own choices. Our main experiments run in observer mode on recorded human-human dialogue, where this complication does not arise. As a secondary analysis, we restrict updates to windows in which a simulated agent channel is silent.
Baselines
The naive reference point is the source Voice Activity Projection model, frozen and evaluated on TurnBench with its standard decision rules. TurnBench reports that this model reaches 0.845 end-of-turn recall and 0.945 interruption recall, the highest of the 14 systems evaluated (TurnBench). We found no reported precision or false-positive figures on Lacuna for it, so we will recompute all metrics ourselves under TurnBench's causal-commit scoring. A silence-threshold voice activity detector, swept over the 500–1000 ms range Schlangen describes, anchors the reactive end of the scale.
The strongest existing source-free method is Tent applied to the same normalization parameters, on the same streams, with the same update schedule and buffer. Its vision results, such as 44.0% top-1 error on ImageNet-C against 49.9% for test-time normalization (Tent), show it is a serious method. We found no reported result on Lacuna for Tent or any test-time adaptation on turn-taking. A normalization-statistics-only variant in the style of UBNA rounds out this family. All adaptive methods receive the same hyperparameter search budget. The search runs on the casual and narrative styles' development portion only, and results are reported on held-out speakers in all six styles.
The ablation that isolates our key ingredient is online threshold recalibration. It uses the identical delayed labels but adjusts only a per-conversation scalar decision threshold, plus a hold-versus-shift prior, so that commit decisions are calibrated against what actually happened. A second ablation, shuffled futures, adapts on delayed labels drawn from a different conversation of the same style. It tests whether gains come from aligning with this conversation's dynamics or merely from shifting style-level base rates. An upper bound is supervised fine-tuning of the same parameters on TurnBench's 104-hour labeled training set for the matching style.
Beating the baselines means two things. The first is a statistically reliable reduction in interruption false-positive rate, at matched or better recall, against both frozen Voice Activity Projection and Tent in at least the casual and argumentative styles. The second is a clear margin over threshold recalibration in onset latency at equal false-positive rate. The latency margin is the signature of a real change in the shape of the model's predictions over time.
Why not just…?
Why not just fine-tune on TurnBench's labeled training set? That is the supervised upper bound, and we expect it to win in-domain. It answers a different question, though. It requires knowing the style in advance, collecting labeled audio for it, and retraining offline, and none of that is available when a deployed agent meets an unfamiliar speaker or register. The relevant measure is how much of the gap between frozen and fine-tuned models delayed self-supervision closes without labels. We will report this as a fraction per style.
Why not just recalibrate the decision threshold online? This is the real threat. TurnBench's picture is a model with high recall and too many false alarms, which is exactly what threshold tuning addresses. That is why recalibration is our central ablation and uses the same delayed labels. If it matches the full method on both false-positive rate and latency, we will conclude that calibration suffices and report that as the finding. If our method wins mainly on latency at equal false-positive rate, that shows adaptation is reshaping the model's predictions, not only moving the line it acts on.
Why not just use Tent, the standard source-free method? Tent's objective pushes predictions toward confidence. At an overlap onset, the honest prediction is uncertain between backchannel and interruption, because the two are acoustically indistinguishable at that moment (TurnBench). We expect Tent to collapse toward whichever reading the source model already favors. We will check this directly by tracking the predicted backchannel rate over time against the delayed-observed rate.
Why not just deploy a larger audio foundation model? The evidence in the file runs the other way. Gemini 3.1 Live manages the floor conservatively, with a median latency of 1234 ms on TurnBench, and Moshi and GPT-4o cascades fail on backchannels and interruptions (Talking Turns). Scale has not solved style sensitivity, and adapting a small predictive head is cheap enough to run per conversation.
Proposed Experiments
The primary experiment runs all methods prequentially over TurnBench's six styles with speaker-disjoint splits. We report end-of-turn and interruption precision, recall, and false-positive rate under the causal-commit rule. We also report median onset latency relative to the human −151 ms lead, and the model's prequential future-state log-likelihood. As a cross-corpus check, we adapt the Switchboard-trained model on Columbia Games and on Fisher.
The decisive checks are three. The first is whether delayed-label adaptation beats Tent on interruption false positives in the casual style, where backchannels are densest. The second is whether it beats threshold recalibration on latency at a matched false-positive rate. The third is whether the shuffled-future control loses most of the gain. A large loss shows the model is tracking this conversation's dynamics rather than style-level base rates.
The contamination probe compares the frozen model's drop between original and timing-preserving acoustic disguises (codec change, pitch shift) on Switchboard versus TurnBench. A larger drop on Switchboard would suggest that part of the model's familiar-corpus skill is memorized acoustics, and that adaptation gains should be reported on fresh audio. Failure modes we will report, not hide, are drift over long persistent adaptation and any style where adaptation hurts.
References
- From reaction to prediction: experiments with computational models of turn-taking
- Show & Tell: Voice Activity Projection and Turn-taking
- TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
- Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
- Response-conditioned Turn-taking Prediction
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- Unsupervised BatchNorm Adaptation (UBNA): A Domain Adaptation Method for Semantic Segmentation Without Using Source Domain Representations
- Few-Shot Adaptation of Pre-Trained Networks for Domain Shift
- Stability and Hypothesis Transfer Learning
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Adaptive Test-Time Training for Predicting Need for Invasive Mechanical Ventilation in Multi-Center Cohorts
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks