Tracking Speaker Identity in Multi-Turn Dialogue Models

238 concepts160 papers4 proposals

When a human reads a dialogue transcript, they do not just process a continuous stream of words; they track a shifting matrix of perspectives. In a standard text document, the narrative voice is relatively stable. In a conversation, however, the ground shifts with every turn. Indexical words like "I" or "you" completely invert their real-world referents depending on who is currently speaking.

If a sequence model processes a dialogue as a flat string of text, it must infer these shifting boundaries and identities purely from implicit semantic cues. As highlighted by Sun et al. in their work on the DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension dataset, standard language models struggle heavily with tracking participant-specific context, creating a massive gap between machine and human conversation comprehension. To bridge this gap, a prominent research direction focuses on explicitly injecting speaker identity and role information into the vector spaces and token streams of conversational models.

Token-Level Disentanglement

The most direct way to break the flat-text illusion is to introduce explicit structural markers into the text stream itself. Instead of feeding raw text turns sequentially, models can be guided by specialized tokens or structured prompts that delineate speaker boundaries.

For instance, Budzianowski et al. integrated special learned tokens into the input sequence of GPT-2 to differentiate between user and system turns in task-oriented dialogues (Hello, It's GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems). By doing so, the model avoids confusing its own previous responses with the user's inputs.

Similarly, Kowsher et al. utilized token-type embeddings to assign a categorical identifier to each token based on its source, demonstrating that this structural map significantly enhances conversational relevance and coherence (Token Trails: Navigating Contextual Depths in Conversational AI with ChatLLM).

Standard Input:  [How can I help you][I need to reset my password][Sure, click here]
Token-Level:     [SYS][How can I help you][USER][I need to reset my password][SYS][Sure, click here]

In more complex multi-party scenarios where the number of interlocutors grows, Tan et al. showed that weaving explicit speaker and addressee metadata directly into prompts transforms a flat text sequence into a structured record, allowing large language models to maintain perspective consistency (Is ChatGPT a Good Multi-Party Conversation Solver?).

Surprisingly, even when dealing with downstream tasks like coreference resolution, abstract flags are often beaten by natural language. Bui et al. found that prepending actual speaker names as natural language tokens within the text text stream allowed the model's self-attention mechanism to directly associate pronouns with their real-world mentions in a unified representational space (End-to-end Neural Coreference Resolution Revisited: A Simple yet Effective Baseline).

Architectural Speaker Embeddings

While token-level modifications are highly effective, they consume valuable context window space and treat speaker identity as just another vocabulary word. A deeper approach integrates speaker identity directly into the model's architecture via specialized embedding layers.

Just as a transformer adds positional embeddings to give tokens spatial awareness, models can add speaker embeddings to give them identity awareness. In SA-BERT, developed by Gu et al., speaker-specific identity vectors are fused directly into the token embeddings (Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots). This ensures that the source of each utterance is tracked at every layer of the network.

$$\text{Embedding} = \text{Token Embedding} + \text{Positional Embedding} + \text{Speaker Embedding}$$

In TurnGPT, Ekstedt et al. introduced a dedicated speaker identity embedding layer to capture "pragmatic completeness"—understanding not just when a sentence is grammatically finished, but when a specific speaker is actually done talking based on the conversation's history.

By embedding this metadata directly into the vector space, the self-attention mechanism can dynamically modulate how tokens interact. For example, a token can attend differently to a historical utterance depending on whether it was spoken by the same participant or an interlocutor. Kim et al. took this further by incorporating a trainable "speaker indicator" vector directly into the attention mechanism, allowing the model to weight past conversation history dynamically based on the relationship between the current speaker's role and previous dialogue participants (Decay-Function-Free Time-Aware Attention to Context and Speaker Indicator for Spoken Language Understanding).

Why It Works: Grounding and Disentanglement

Why does giving a model explicit awareness of who is speaking yield such substantial performance gains across dialogue tasks? The intuition relies on two primary mechanisms: deictic grounding and thread disentanglement.

Grounding Deictic and Role-Based Meaning

In many environments, the functional weight of a statement depends entirely on the role of the speaker. For instance, in legal judgment prediction, Ma et al. demonstrated that mapping utterances to different vector spaces based on whether the speaker is a judge, plaintiff, or defendant is critical for capturing the true intent of an adversarial dialogue (Legal Judgment Prediction with Multi-Stage Case Representation Learning in the Real Court Setting). A statement of fact from a defendant has a completely different legal implication than the same statement from a judge.

Similarly, in multi-party character linking, Bai et al. showed that integrating speaker identity embeddings into mention representations allows models to resolve ambiguous references like "I" or "you" by grounding them firmly in the conversational context (Joint Coreference Resolution and Character Linking for Multiparty Conversation). The embedding provides a constant anchor, preventing the identity of "I" from dissolving as the conversation switches hands.

Disentangling Multi-Party Noise

When multiple agents speak, a flat transformer struggles to isolate individual conversational threads. By using agent-aware representations, as proposed by Ouchi et al., models can filter out background noise in crowded threads by associating specific utterances with the unique historical states of individual agents (Addressee and Response Selection for Multi-Party Conversation).

This architectural awareness allows the network to track two distinct dimensions of dialogue simultaneously: the global, inter-speaker flow of the conversation, and the internal, intra-speaker consistency of a single participant's persona or logic (Automatic Evaluation of Neural Personality-based Chatbots). When the attention mechanism can group tokens by speaker identity, it can easily trace a single speaker's train of thought across dozens of intervening turns.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 175 papers
  2. Version 1 · 2026-10-04 · 280 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_speaker_aware_context_modeling_17e89744,
  title = {speaker-aware context modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speaker-aware-context-modeling/rdr_17e897449094459ea7134a45fd0183fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_speaker_aware_context_modeling_17e89744_v2,
  title = {speaker-aware context modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/speaker-aware-context-modeling/rdr_17e897449094459ea7134a45fd0183fb/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake