Modeling Who Says What to Whom in Multi-Party Conversations

This direction is merged into another direction: addressee ambiguity. Its last version stays readable and citable here.

Version 1 of 116 papers17 conceptsfrozen 2026-10-04

Most conversational AI is built on a simple assumption: there are two people talking, and they are talking to each other. In this dyadic world, the "who" and the "whom" are trivial. However, real-world human interaction is rarely so tidy. Whether it’s a family dinner, a business meeting, or a group chat, we constantly navigate Multi-Party Conversations (MPC) where the intended recipient of a message is often left unstated.

Researchers are moving beyond simple sequence-to-sequence models to solve the "Who-What-Whom" puzzle—the challenge of identifying the speaker, the content, and the addressee in a crowded social field.

The Addressee Ambiguity Problem

In a two-person dialogue, if I speak, I am the speaker and you are the addressee. But as Jia-Chen Gu and Zhen-Hua Ling point out in Learning WHO Saying WHAT to WHOM in Multi-Party Conversations, group settings introduce a "final boss" level of complexity. Participants rarely name their addressees explicitly. We rely on context, social dynamics, and visual cues to know who is being spoken to.

For a robot or an AI, this lack of explicit markers creates a "contextual drift." If a system doesn't know who a previous message was directed at, it cannot determine which parts of the conversation history are relevant to the current turn. This makes generating a coherent response nearly impossible using standard models.

From Sequences to Graphs

To solve this, researchers have shifted from treating conversation as a linear string of text to treating it as a dynamic graph. Hiroki Ouchi and Yuta Tsuboi pioneered the Addressee and Response Selection (ARS) task in Addressee and Response Selection for Multi-Party Conversation, proposing that a system must track the evolving state of every participant simultaneously.

Modern frameworks like MADNet, developed by Jia-Chen Gu and colleagues in MADNet: Maximizing Addressee Deduction Expectation for Multi-Party Conversation Generation, use heterogeneous graphs to map these relationships. By modeling the "graphical" flow of information, these systems can better understand how different threads of conversation interweave.

Deducting the Latent Addressee

Since real-world data often lacks "addressee labels" (we don't tag our friends in every sentence we speak), researchers have turned to the Expectation-Maximization (EM) framework. The intuition here is to treat the addressee as a latent variable—a hidden piece of information that the model must guess.

In EM Pre-training for Multi-party Dialogue Response Generation, Yiyang Li demonstrates that a model can iteratively improve its understanding by "deducing" who is being addressed and then using that deduction to refine its language generation. This allows models to learn from massive amounts of unannotated group chat data.

The Role of Multi-modal Cues

In physical spaces, the "whom" isn't just in the text; it's in the body. The Vernissage Corpus, introduced by Dinesh Babu Jayagopi and Jean-Marc Odobez in THE VERNISSAGE CORPUS: A MULTIMODAL HUMAN-ROBOT-INTERACTION DATASET, highlights how robots must use head-pose estimation and tracking to distinguish between a person talking to them versus a person having a "side conversation" with another human.

Similarly, Thao Le Minh and Koichi Shinoda showed in Deep Learning Based Multi-modal Addressee Recognition in Visual Scenes with Utterances that combining visual cues like eye gaze with textual content significantly outperforms text-only models. To further this, Yueqian Wang and others recently released Friends-MMC (Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding), a dataset based on the TV show Friends that challenges AI to identify speakers and addressees in complex, situated visual scenes.

Can LLMs Solve the Puzzle?

While Large Language Models (LLMs) like ChatGPT are powerful, they are primarily trained on dyadic-style data. Chao-Hong Tan and Jia-Chen Gu explored this in Is ChatGPT a Good Multi-Party Conversation Solver?. They found that while LLMs struggle with the "who-said-what-to-whom" nightmare out of the box, a technique called MPC Structure Incorporation—explicitly weaving speaker and addressee metadata into the prompt—can significantly boost their performance.

Ultimately, the goal of this research direction is to move AI from being a polite one-on-one interlocutor to a socially intelligent participant capable of following the complex, multi-directional flow of human life.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_who_says_what_to_whom_in_multi__ae94a59d,
  title = {Modeling Who Says What to Whom in Multi-Party Conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-who-says-what-to-whom-in-multi-party-conversations/rdr_ae94a59dcabc4c789661f6c2fb6ad2a8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_who_says_what_to_whom_in_multi__ae94a59d_v1,
  title = {Modeling Who Says What to Whom in Multi-Party Conversations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-who-says-what-to-whom-in-multi-party-conversations/rdr_ae94a59dcabc4c789661f6c2fb6ad2a8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}