Modeling Inter Speaker Influence in Dialogue

79 concepts60 papers1 proposal

Traditional natural language processing often treats conversation as a sequence of isolated text snippets. If you want to predict how a speaker feels, standard models look at their current utterance or perhaps their immediate personal history. However, human conversation is fundamentally a co-constructed, bi-directional dance. When two people talk, they do not operate in vacuums; their emotional states constantly collide, mirror, and reshape one another.

To truly understand dialogue, machine learning models must transition from individual-centric analysis to frameworks that capture inter-speaker influence. This paradigm shifts the focus from what is being said to how interlocutors dynamically alter each other's internal states over time.

The Two Core Forces: Inertia vs. Influence

At the heart of dyadic (two-person) interaction modeling lies a tension between two distinct psychological forces:

  1. Emotional Inertia (Intra-speaker dependency): The tendency for an individual's emotional state to persist from one turn to the next. If a speaker is angry, they are highly likely to remain angry in their next utterance unless an external force disrupts their mood.
  2. Inter-speaker Influence (Interpersonal dependency): The emotional shift triggered directly by the conversation partner's behavior or emotional expression. This force drives social phenomena like empathy, mimicry, or the rapid escalation of a conflict.
[Speaker A: Turn t-1] --(Inertia)--> [Speaker A: Turn t+1]
         \                                  ^
          \                                /
     (Influence)                      (Influence)
            \                              /
             v                            /
        [Speaker B: Turn t] -------------/

As shown by Devamanyu Hazarika et al. in their work on the Conversational Memory Network (CMN), explicitly separating these two forces yields a substantial performance boost in emotion recognition. Instead of blending all context together into a single vector, modern architectures maintain distinct memory tracks for each speaker's internal history while allowing a cross-talk mechanism to simulate interpersonal influence.

Weizhou Shen et al. validated this distinction empirically with DialogXL, demonstrating that removing specialized mechanisms for tracking individual speaker history versus listener responses leads to a significant decline in predictive accuracy. Similarly, the Emotion Interaction Network (DialogueEIN) proposed by Yuchen Liu et al. formalizes this balance by explicitly modeling the interplay between internal "emotional inertia" and external "emotional stimuli" (DialogueEIN: Emotion Interaction Network for Dialogue Affective Analysis).

Architectural Paradigms for Tracking the Feedback Loop

How do we mathematically formalize this back-and-forth? Researchers have explored several architectural families to capture these dependencies:

  • Dynamic Bayesian Networks (DBNs): Early foundational work by Chi-Chun Lee et al. used graphical models to represent conversation as a sequence where a speaker's current state depends on both their own previous state and the previous state of their interlocutor (Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions). This provided an elegant, probabilistic way to model the "cross-talk" between two internal states.
  • Recurrent Memory Networks: To handle multimodal data (audio, video, and text), Devamanyu Hazarika and Soujanya Poria developed ICON, which uses a hierarchical GRU structure and multi-hop memory to track the historical trajectories of both participants, allowing the model to navigate complex emotional mirroring.
  • Cross-Attention and Transformers: In modern architectures, self-attention is augmented with cross-attention to map the target speaker's state against the interlocutor's timeline. For instance, Jinming Zhao et al. introduced the Attentive Interaction Model (AIM), which uses attention to weigh how strongly an interlocutor's history triggers emotional changes in the primary speaker (Speech Emotion Recognition in Dyadic Dialogues with Attentive Interaction Modeling).

The Listener as a Mirror

One of the most profound insights from this line of research is that the listener's reaction is often a cleaner signal of the speaker's emotion than the speaker's own words.

In LaERC-S, Yumeng Fu et al. leveraged Large Language Models to reason about speaker characteristics, finding that prioritizing the inferred reaction of the listener provides a clearer reflection of the underlying emotional context than the speaker's own utterances. This social feedback loop is critical because human communication is inherently reciprocal; our social strategies are often mirrored or prompted by our partner (Ran Zhao et al. in Automatic Recognition of Conversational Strategies in the Service of a Socially-Aware Dialog System).

This dyadic mirroring extends beyond text into multimodal behavior:

Why It Works: Co-Construction and Consensus

When we model conversation as a joint state rather than a sequence of independent actions, we unlock more natural AI behaviors. For example, Lei Shen et al. demonstrated that modeling empathy as a shared emotional consensus—a convergence of emotional understanding between two speakers—leads to dialogue generation that is significantly more contextually accurate and emotionally resonant (Constructing Emotional Consensus and Utilizing Unpaired Data for Empathetic Dialogue Generation).

Furthermore, in generative tasks like creating 3D avatars, models trained on multi-person interaction data produce far more responsive non-verbal expressions (such as nodding or smiling) than those trained on single-person monologues. As explored by Minh Tran 0004 et al. with DIM, capturing the interdependence of conversational partners is necessary to avoid static or unnatural behaviors.

Ultimately, dyadic interaction modeling shifts the paradigm from passive observation to active, relational context. By treating conversation as a bi-directional feedback loop, machines can move past simple snapshot classifications and begin to understand the true relational dynamics of human interaction.

Key Foundational Work

Emotion Estimation Algorithm based on Emotion Occurrence Sentence Pattern (2007) — contributed 2 concept elements to this direction.
Detecting user engagement in everyday conversations (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dyadic_interaction_modeling_63a55707,
  title = {dyadic interaction modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dyadic-interaction-modeling/rdr_63a557075d614c9a9595c3eac92729f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dyadic_interaction_modeling_63a55707_v1,
  title = {dyadic interaction modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dyadic-interaction-modeling/rdr_63a557075d614c9a9595c3eac92729f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake