Self- and Pseudo-self-supervised Prediction of Speaker and Key-utterance for Multi-party Dialogue Reading Comprehension

Yiyang Li, Hai Zhao

Abstract

Multi-party dialogues present a significant challenge for Machine Reading Comprehension (MRC) due to shifting speaker roles and irrelevant conversational noise. This paper proposes a novel approach using two auxiliary tasks—speaker prediction and key-utterance identification—to enhance model understanding without requiring manual annotations. By decoupling these tasks from the main encoding process, the model effectively learns conversational structure and filters noise, achieving state-of-the-art performance on the FriendsQA and Molweni benchmarks.

The Challenge of Multi-party Dialogue

Standard text analysis often struggles with the dynamic nature of multi-party conversations. In datasets like FriendsQA, which uses transcripts from the TV show Friends, the meaning of pronouns like "I" or "you" shifts constantly depending on the speaker and the intended listener. This creates an "intricate speaker information flow" that is difficult for models to track. Additionally, long dialogues are often cluttered with "fluff"—utterances that provide social context but are irrelevant to answering specific questions.

Example of a multi-party dialogue from FriendsQA showing complex speaker interactions and associated questions.

Previous attempts to solve this relied on complex, manually-defined graph modules or expensive human-annotated discourse labels. The authors aimed to teach a model these social and structural nuances using only the data already present in the text, creating a "labour-free" solution.

Methodology: Decoupling Speaker and Key-utterance Information

The authors propose a model architecture featuring two specialized "Decoupling Blocks" added on top of a standard Transformer encoder (such as BERT or ELECTRA):

  1. Speaker Information Decoupling Block (SIDB): This uses a self-supervised task where the model predicts a randomly masked speaker's name. To succeed, the model must understand the conversational flow and resolve coreferences. For instance, if Monica says "Tell him," and the next speaker says "No," the model learns to identify the second speaker as Rachel based on the interaction history.
  2. Key-utterance Information Decoupling Block (KIDB): This uses a pseudo-self-supervised task. During training, the model identifies which utterances contain the answer (the "ground truth"). It trains a sub-module to predict these "key" utterances, effectively learning to prioritize relevant information and ignore conversational noise.
The proposed model architecture featuring Speaker and Key-utterance Information Decoupling Blocks (SIDB and KIDB) and the detaching operation.

A critical technical detail is the "Detaching Operation." To prevent the model from "cheating" by seeing speaker names during initial encoding, representations are detached from the main computational graph. A specific attention mask is used to prevent label leakage while still allowing the model to develop deep, speaker-aware features.

Experimental Evaluation

The model was evaluated on two major multi-party dialogue datasets:

  • FriendsQA: Contains over 10,000 questions based on transcripts from the first four seasons of Friends, characterized by colloquial language and complex character interactions.
  • Molweni: Derived from the Ubuntu Chat Corpus, focusing on technical discussions and including unanswerable questions to test the model's ability to handle domain-specific jargon.

The proposed method set new state-of-the-art (SOTA) records. On FriendsQA, using an ELECTRA-based backbone, the model achieved 55.8 Exact Match (EM) and 72.3 F1 score. On Molweni, it reached 49.2 EM and 64.0 F1, notably outperforming models that relied on additional manual discourse annotations.

Key Insights

The research demonstrates that explicit, human-designed graphs are not strictly necessary for modeling dialogue. Instead, auxiliary tasks allow the model to implicitly learn the underlying structure. The SIDB module particularly improved performance on "Who" questions, confirming its effectiveness in character tracking. Furthermore, the model's advantage over baselines grew as the number of speakers and utterances increased, proving its robustness in complex social environments. Most importantly, these gains are achieved without any extra manual labeling.

Conclusion and Future Directions

While the model significantly advances dialogue MRC, speaker prediction remains a challenging task for AI compared to human intuition. Future work may involve deeper integration of discourse structures, such as "reply-to" relationships, and applying these decoupling techniques to large-scale generative models to enhance conversational consistency in dialogue agents.

Summary updated

BibTeX
@inproceedings{li2021self,
  author = {Yiyang Li and Hai Zhao},
  title = {{Self- and Pseudo-self-supervised Prediction of Speaker and Key-utterance for Multi-party Dialogue Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.176},
}
Figure