When two humans engage in conversation, the transition between speakers is remarkably seamless. The typical gap between turns is roughly 200 milliseconds—a window so brief that it undercuts basic human reaction time for speech production. If listeners simply waited for a speaker to stop talking before planning their response, conversations would be plagued by awkward, multi-second silences.
To achieve this fluid exchange, the human brain does not react to silence; it predicts it. Listeners continuously track acoustic trajectories—specifically prosodic features like pitch, intensity, and rhythm—to project exactly when a speaker's turn will reach a Transition Relevance Place (TRP).
The Acoustic Anatomy of Turn Signaling
How does a speaker structurally signal that they are about to yield or hold the floor? While syntax and semantics provide structural boundaries, prosody acts as the real-time coordinator.
In a foundational study on predictive turn-taking, David Schlangen demonstrated in From reaction to prediction: experiments with computational models of turn-taking that computational models can achieve human-like prediction accuracy by focusing heavily on the acoustic cues embedded in the final word of an utterance. Specifically, a simultaneous decline in both fundamental frequency ($F_0$) and intensity serves as a robust physical signal that a speaker is ready to yield the floor.
Beyond pitch and loudness, the temporal structure of the words themselves stretches as a turn nears completion. This phenomenon, known as "final lengthening," involves the elongation of speech sounds immediately preceding a pause. In Measuring final lengthening for speaker-change prediction, Anna Hjalmarsson et al. captured this stretching effect automatically using Mel-spectral flux, proving that tracking these subtle spectral shifts over time provides a highly reliable indicator of an upcoming speaker change.
Turn-Yielding Trajectory:
Pitch (F0) ───↘
Intensity ───↘
Syllables ───📋───📋───📋──────📋 (Final Lengthening)
│
▼
[Predicted Turn End]
Conversely, prosody can also be used to invite the listener to participate without yielding the primary turn—a mechanism known as backchanneling (e.g., saying "uh-huh" or "mhm"). Agustín Gravano et al. discovered in Backchannel-inviting cues in task-oriented dialogue that specific rising intonation contours (such as H-H% and L-H%) combined with elevated pitch and intensity act as a clear invitation for listener feedback.
When listeners do interject with a backchannel, they unconsciously match the speaker's acoustic space; Mattias Heldner et al. showed in Pitch similarity in the vicinity of backchannels that backchannels exhibit a high degree of pitch similarity to the preceding utterance, allowing them to remain supportive and unobtrusive.
Temporal Buffering: Projecting the Horizon
If turn-taking cues were only present in the final milliseconds of an utterance, listeners would still lack the time required to formulate a response. The predictive mechanism must operate further upstream, providing a "temporal buffer."
Research shows that speakers embed these prosodic presignals deep within their utterances. Tomoko Ohsuga et al. demonstrated in Investigation of the relationship between turn-taking and prosodic features in spontaneous dialogue that even when the final segment of an utterance is entirely removed, models can still predict upcoming turn changes with high accuracy based on the pitch and loudness slopes occurring much earlier.
This long-range signaling is further supported by Kikuo Maekawa in A New Model of Final Lowering in Spontaneous Monologue, which revealed that the acoustic down-drift signaling utterance finality actually begins as early as the penultimate accentual phrase.
Timeline of an Utterance:
[ Penultimate Phrase ] ──► Pitch lowering begins (Early Presignal)
[ Last Prominent Word ] ──► Anchor point for TRP projection
[ Final Word ] ──► Syllable lengthening & intensity drop
[ Boundary ] ──► Seamless Speaker Switch (~200ms gap)
These early acoustic markers act as structural guideposts. Rob J. J. H. van Son et al. explored this in Prominent words as anchors for TRP projection, finding that prominent words within an utterance serve as "anchors" that allow listeners to project the exact timing of a TRP. Crucially, the variance in human response times is dictated by the positioning of these anchor points rather than the cognitive processing duration, meaning our conversational timing is structurally locked to the rhythmic anchors laid down by the speaker.
The Autonomy of Prosody
One of the most compelling reasons this system works so efficiently is that prosody operates largely independently of semantic comprehension. The brain can parse the timing of a conversation without needing to fully decode the meaning of the words first.
This was elegantly demonstrated by Agustín Gravano et al. in Who Do You Think Will Speak Next? Perception of Turn-Taking Cues in Slovak and Argentine Spanish. When participants were exposed to completely de-lexicalized, masked, or hummed audio—or even speech in a language they did not understand—they remained remarkably adept at predicting whether a speaker would hold the floor or yield it.
Similarly, word-final intonation contours remain a highly persistent, "hard-coded" signal for communicative intent even when surrounding contextual cues are altered, as shown by Agustín Gravano et al. in On the role of context and prosody in the interpretation of 'okay'.
This decoupling of sound and meaning creates a powerful shortcut for conversational AI systems. Traditional spoken dialogue systems suffer from high latency because they rely on a linear pipeline:
$$\text{Audio} \longrightarrow \text{Speech-to-Text (ASR)} \longrightarrow \text{Language Model} \longrightarrow \text{Text-to-Speech}$$
By the time the ASR transcribes the final word, the natural 200ms window has already closed.
Because prosody is autonomous, developers can bypass this bottleneck entirely. Olac Fuentes et al. demonstrated this in A Filter-Based Approach to Detect End-of-Utterances from Prosody in Dialog Systems by applying fast, constant-time speech filters directly to raw audio. This approach successfully detected end-of-utterance boundaries without requiring any linguistic transcription.
This acoustic-first paradigm is highly complementary to language modeling. As Matthew Roddy et al. noted in Neural Generation of Dialogue Response Timings, acoustic features excel at pinpointing the exact millisecond a phrase finishes, whereas linguistic features are best suited for determining the appropriate length of the subsequent silence based on semantic context.
Continuous Voice Activity Projection
Early computational models treated turn-taking as a discrete classification problem evaluated at fixed intervals. Modern architectures, however, treat turn-taking as a continuous forecasting task.
A prominent framework in this space is Voice Activity Projection (VAP), introduced by Erik Ekstedt et al. in Show & Tell: Voice Activity Projection and Turn-taking. Instead of predicting a binary "end of turn" state, VAP models continuously project the future voice activity of multiple speakers simultaneously onto a shared probabilistic horizon.
VAP Continuous Projection Horizon:
Speaker A: ▇▇▇▇▇▇▇▇░░░░░░░░░░░░░░░░ (Probability of speaking dropping)
Speaker B: ░░░░░░░░░░░░░▇▇▇▇▇▇▇▇▇▇▇▇ (Probability of speaking rising)
────────────────────────►
Time (Continuous)
By training via self-supervision on raw conversational audio, these models implicitly learn to map complex, multi-layered prosodic trajectories directly to future conversational states.
When fine-tuning these continuous projection models for highly sensitive tasks like real-time backchannel prediction, Koji Inoue et al. discovered in Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection that acoustic intensity dynamics (volume changes) often exert a heavier influence on timing than pitch variations. When pitch contours are flattened experimentally, models still successfully leverage volume-related cues to pinpoint the exact moment a listener should provide feedback.
Ultimately, prosodic turn-taking prediction works because human speech is physically structured to prevent conversational collisions. Long before a speaker stops vocalizing, their voice leaves behind a trail of acoustic breadcrumbs—declining pitch, fading intensity, and elongated syllables—allowing the listener's brain to map the trajectory, anticipate the finish line, and step into the conversation without missing a beat.
Key Foundational Work
Prominent words as anchors for TRP projection (2006) — contributed 7 concept elements to this direction.
Investigation of the relationship between turn-taking and prosodic features in spontaneous dialogue (2005) — contributed 6 concept elements to this direction.
Timing of experimentally elicited minimal responses as quantitative evidence for the use of intonation in projecting TRPs (2005) — contributed 4 concept elements to this direction.