Addressee Detection in Multi-Party Conversation

23 concepts21 papers1 proposal

In a traditional two-party dialogue, the question of who is being spoken to is trivial: the recipient is always the other person. However, as conversational AI moves from isolated "phone booth" interactions into the real world, this assumption collapses. In "open-world" environments—such as a space station, a hospital clinic, or a shared living room—multiple people may be talking to each other, to a system, or even to themselves simultaneously.

The challenge of identifying the intended recipient of an utterance is known as addressee detection. Without solving this, systems suffer from the "open mic" problem, where they mistakenly respond to side-talk or ignore direct commands.

The Open Mic Problem

The fundamental difficulty in multi-party conversation (MPC) is that utterances often lack explicit markers of their target. As Aist et al. observed in early work on intelligent assistants for the International Space Station, a system that is "always listening" cannot easily distinguish between a command directed at the assistant and general crew-to-crew conversation.

This ambiguity creates a "who-says-what-to-whom" nightmare for automated systems. Bohus et al. characterize this as a source association problem: in unconstrained environments, an agent must differentiate between directed social signals and ambient noise or side conversations. If a system fails to resolve this, it cannot maintain a coherent dialogue state because it doesn't know which parts of the conversation history are relevant to the current turn (Gu et al.).

Multimodal Cues for Intent

To resolve this ambiguity, researchers have looked beyond the literal text of an utterance to the physical and acoustic signals that accompany human speech.

One of the most intuitive signals is gaze. Oh et al. proposed "look-to-talk" interfaces, where a user’s eye contact with a software agent serves as a natural trigger for interaction, replacing cumbersome buttons or wake words. Similarly, Jayagopi et al. utilized head-pose estimation and visual tracking to help social robots distinguish between direct engagement and side-talk in group settings.

Acoustics also provide vital clues. Stolcke et al. found that humans often adopt a specific "sing-songy" rhythmic pattern—a unique energy contour—when addressing machines compared to other humans. By combining these prosodic features with linguistic cues, the authors significantly reduced false activations in multi-user environments.

Modeling Latent Social Structure

When physical cues are unavailable—such as in text-based chat or when sensors are limited—systems must rely on the structure of the conversation itself.

Recent approaches treat the addressee as a latent variable that must be inferred to make sense of the dialogue. Li et al. and Gu et al. have developed Expectation-Maximization (EM) frameworks that iteratively "guess" the missing addressee labels in a transcript to improve the generation of context-aware responses. By deducing who is talking to whom, these models can filter out irrelevant side-talk and focus on the correct conversational thread.

Furthermore, the social context of the group provides predictive power. Hajishirzi et al. demonstrated that a "Group-History" model, which tracks the real-time behavior of all peers rather than just the speaker, is significantly more effective at identifying addressees in complex interactions, such as children playing in groups.

Addressee Awareness in Large Language Models

Even state-of-the-art Large Language Models (LLMs) struggle with the many-to-many information flows of multi-party settings. Tan et al. found that while models like GPT-4 are powerful, their performance in MPC tasks is inconsistent unless they are provided with explicit speaker and addressee metadata—a technique termed MPC Structure Incorporation.

This suggests that addressee detection is not just a preprocessing step, but a core component of social intelligence. In fact, Lee et al. showed that forcing a model to identify the intended recipient as an auxiliary task actually improves its ability to perform higher-level reasoning, such as analyzing collaborative competency. By resolving the "who," the model gains a clearer picture of the "what" and "why" of the interaction.

Key Foundational Work

Talking through Procedures: An Intelligent Space Station Procedure Assistant (2003) — contributed 1 concept elements to this direction.
Evaluating look-to-talk (2002) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_addressee_ambiguity_b557bb2d,
  title = {addressee ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/addressee-ambiguity/rdr_b557bb2d028a41948ee553b53548ff5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_addressee_ambiguity_b557bb2d_v1,
  title = {addressee ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/addressee-ambiguity/rdr_b557bb2d028a41948ee553b53548ff5f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake