Modeling the Who-What-Whom of Multi-Party Conversations

This direction is split into other directions: dyadic interaction constraint, multi-party dialogue complexity. Its last version stays readable and citable here.

Version 1 of 1327 papers367 conceptsfrozen 2026-10-04

Most conversational AI is built on a simple assumption: there are two people talking to each other. In this "dyadic" world, the context is linear, and the addressee is always the other person. However, real-world communication—from Discord servers and Slack channels to family dinners and business meetings—is rarely so simple.

Researchers are now moving beyond one-on-one exchanges to model Multi-Party Conversations (MPC), or "polylogues." This shift requires AI to solve a much harder structural problem: tracking not just what is being said, but who is saying it to whom, and how those interactions ripple across a group.

The Multi-Party Hangover

While Large Language Models (LLMs) excel at following a two-person dialogue, they often struggle with the structural "who-to-whom" dynamics of a group. Nicolò Penzo and Bruno Lepri describe this as a Multi-Party Hangover, where models lose track of the conversation's map as the number of participants grows.

In a group, participants constantly shift roles. You might be the sender of one message, the intended addressee of another, and a passive observer (or "overhearer") for the rest. To navigate this, models like the Speaker Interaction RNN (SI-RNN), proposed by Rui Zhang and Honglak Lee, explicitly model these roles. Instead of treating the conversation as a single stream of text, they update speaker states in a role-sensitive way, allowing the AI to predict both the next response and its most likely recipient simultaneously.

Capturing Social Glue and Contingent Behaviors

In a group, people don't just exchange information; they react to the social presence of others. This is known as contingent behavior—the subtle ways our speech and gestures are influenced by the collective group rather than just the last person who spoke.

To capture this "social glue," Dong Won Lee and Cynthia Breazeal introduced MultiPar-T, a transformer architecture featuring Crossperson Attention MultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations. This mechanism allows the model to attend to simultaneous data streams from multiple participants, identifying how one person's engagement level might be triggered by another's non-verbal cues.

Similarly, in complex environments like Dungeons & Dragons, an AI must use a "Theory-of-Mind" to guide players. Pei Zhou and Xiang Ren developed the G4C framework, which enables an AI Dungeon Master to mentally simulate how different players will react to a statement before saying it. This intentional, goal-oriented guidance is essential for collaborative tasks where the AI must manage multiple human personalities at once.

Disentangling the Conversation Web

One of the greatest hurdles in multi-party research is "entanglement." In a busy IRC chat or a large meeting, multiple topics often run in parallel, creating a messy web of overlapping threads. Standard models that rely on "next-utterance prediction" fail here because the "next" message in a log might belong to a completely different sub-conversation.

Jonathan K. Kummerfeld and colleagues addressed this by releasing a massive dataset for conversation disentanglement. By manually annotating thousands of messages to show which ones actually respond to which, they've enabled models to treat conversations as multi-directional graphs rather than linear sequences. This allows an AI to "unweave" the threads of a group chat to understand the specific context of each sub-group.

From Chatbots to Teammates

The ultimate goal of this research is to move AI from being a 1-on-1 assistant to a functional team member. This requires a deep understanding of mutuality of belief—tracking not just what the AI knows, but what it believes is shared knowledge among the entire group. This concept, explored as early as 1993 in Air Traffic Control simulations, remains the cornerstone for modern collaborative agents.

Whether it's a social robot in a geriatric hospital managing interactions between patients and doctors, or an AR system providing real-time social suggestions during a party, the future of conversational AI lies in its ability to navigate the complex, non-linear, and deeply social world of the group.

Papers in this version

Showing 60 of 327 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_the_who_what_whom_of_multi_part_037fc811,
  title = {Modeling the Who-What-Whom of Multi-Party Conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-the-who-what-whom-of-multi-party-conversations/rdr_037fc8115c1945d5b68bf1bbc2d3b2cd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_the_who_what_whom_of_multi_part_037fc811_v1,
  title = {Modeling the Who-What-Whom of Multi-Party Conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-the-who-what-whom-of-multi-party-conversations/rdr_037fc8115c1945d5b68bf1bbc2d3b2cd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}