While modern Large Language Models (LLMs) can generate remarkably fluent responses, they frequently suffer from what researchers call an "identity crisis" in multi-party conversations. They can capture the "what" of a discussion—the topics, the emotions, and the facts—but they often fail at the "who." This failure to maintain a stable internal map of participants leads to speaker-action misattribution, where a model correctly identifies an event but assigns it to the wrong person.
The Logical Gap Between Fluency and Attribution
Linguistic fluency does not guarantee a robust representation of conversational logic. As Shuaijie She and colleagues demonstrate in Exploring the Dialogue Comprehension Ability of Large Language Models, even state-of-the-art models like ChatGPT fail to be factually consistent in roughly 16% of summaries, primarily due to swapping subjects and objects or omitting participants entirely.
This "who did what to whom" problem is particularly acute in multi-turn dialogues. Models often struggle with subject-object swaps, where the roles of the speaker and the listener are inverted. For instance, in a medical context, a model might correctly extract a symptom but attribute it to the doctor instead of the patient. Yuan Xia and the team behind the SAFE framework argue that even speaker-aware pre-training is insufficient; models require explicit mechanisms to model the complex relationships between different parts of a conversation to avoid these critical failure modes.
Why Participant Tracking Fails
The difficulty of tracking participants stems from several conversational dynamics:
- Coreference and Pronouns: Conversations are dense with pronouns and informal language. Xiangru Tang notes in CONFIT that nearly half of factual errors in dialogue summarization involve misattributing information, often because models lose the thread of who a pronoun refers to across multiple turns.
- Distractor Entities: In tasks like quote attribution, the presence of other names in the surrounding text often misleads models. Wenjie Zhong and Hiroya Takamura found that models struggle to distinguish the actual speaker from other entities mentioned in the context, even when the speaker's name is explicitly nearby.
- Persona Leakage: In retrieval-based chatbots, a lack of clear differentiation between the bot's persona and the user's persona can lead to "persona leakage." Jia-Chen Gu shows in Partner Matters! that if the two personas are not processed distinctly, the system may incorrectly attribute the user's traits or experiences to itself.
The Propagation of Diarization Errors
In real-world applications, the problem begins before the text reaches the NLP model. Automatic Speech Recognition (ASR) systems often struggle with speaker diarization—the process of determining "who spoke when."
Research by Nikolaos Flemotomos in the domain of psychotherapy evaluation reveals a surprising pattern: errors in speaker diarization have a more significant negative impact on behavioral coding accuracy than actual word-level transcription errors. If a system misattributes a therapist's empathetic statement to the patient, the entire quality metric for that session becomes invalid. This challenge is echoed in the work of Abhinav Ramesh Kashyap, whose MEDSAGE framework uses LLMs to generate synthetic "noisy" dialogues to help models become more robust to these inevitable diarization failures.
Architectural Solutions for Speaker Awareness
To solve these issues, researchers are moving beyond simple text concatenation toward "speaker-aware" architectures:
- Explicit Structural Markers: Deyi Xiong and Jun Quan demonstrate that simple utterance tagging—explicitly labeling each turn with a speaker ID—significantly improves dialogue state tracking by helping the model distinguish between user requirements and system prompts.
- Heterogeneous Graph Networks: Models like the Dialogue Heterogeneous Graph Network (D-HGN), proposed by Xiachong Feng, treat speakers, utterances, and concepts as distinct nodes in a graph. This allows the model to explicitly learn the "subtext" and relational dynamics of the conversation.
- Post-Correction Pipelines: Rather than trying to get it right the first time, some researchers use a second pass. Dongyub Lee and colleagues developed a speaker-focused post-correction pipeline that specifically scans generated summaries for swapped, added, or omitted names, significantly improving factual accuracy.
High-Stakes Implications
The necessity of robust participant tracking is most evident in specialized domains. In legal analysis, Elliott Ash points out that a lack of speaker attribution prevents models from distinguishing between a judge's normative rulings and a witness's reported testimony. In mental health, Bichen Wang notes that suicide risk detection systems often fail because they cannot differentiate between a user's self-disclosure and their discussion of someone else's behavior.
Ultimately, the goal of this research direction is to move AI from being a "fluent listener" to a "socially aware participant" that understands the conversation not just as a sequence of words, but as a structured interaction between distinct identities.