Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog

Kaixin Ma, Tomasz Jurczyk, Jinho D. ChoiNAACL 2018 · OpenReview

Introduction

While machine reading comprehension (RC) has seen significant progress on structured, formal texts like news articles and literature, informal multiparty dialog remains a challenging frontier. Human communication in group settings—such as Slack threads, TV scripts, or group chats—is characterized by colloquialisms, repetitions, and information spread across numerous short utterances. This paper addresses the "genre gap" between these informal conversations and the formal, concise summaries typically used to describe them.

The Passage Completion Task

The authors introduce the Passage Completion task for multiparty dialog. In this setting, a model is provided with a transcript and a factual summary where character names have been replaced with blanks (variables). The goal is to correctly identify the character being discussed. This task is particularly difficult because a single fact in a summary might be "diluted" across several turns of small talk in the dialog.

Example of a multiparty dialog, generated passages, and the resulting fill-in-the-blank queries.

Proposed Architecture: CNN+LSTM with Hierarchical Attention

To bridge the gap between colloquial dialog and formal reporting, the researchers developed a hybrid deep learning architecture. This model combines Convolutional Neural Networks (CNNs) for local feature extraction with Long Short-Term Memory (LSTM) networks for sequence modeling.

The core innovation is a two-tier hierarchical attention mechanism:

  1. Utterance-level Attention (UA): Focuses on local word-to-word similarities between the query and individual speaker turns.
  2. Dialog-level Attention (DA): Provides a global view by using 1D convolutions to match the query against the entire sequence of utterances, capturing the "flow" of the conversation.
The proposed CNN+LSTM architecture with hierarchical attention for processing multiparty dialog.

By encoding utterances separately before passing them into a bidirectional LSTM, the model preserves the conversational structure—who said what and when—rather than treating the transcript as a single flat string.

Dataset Construction

The researchers created a new corpus based on the TV show Friends, consisting of:

  • Source Material: 1,681 scenes from all ten seasons.
  • Passages: 4,646 passages derived from fan-site plot summaries and refined via Amazon Mechanical Turk.
  • Queries: 13,487 "fill-in-the-blank" questions.
  • Anonymization: All character names are replaced with generic entity IDs (e.g., @ent01) to ensure the model relies on contextual reasoning rather than memorized character popularity.

Experimental Results

The proposed CNN+LSTM+UA+DA model achieved an accuracy of 72.42%, significantly outperforming standard bidirectional LSTM baselines.

Key findings include:

  • Robustness to Length: While standard models degrade as dialogs grow longer, the hierarchical attention model remains stable. On dialogs lengthened to 100 utterances, it showed a 13.0%+ improvement over the Bi-LSTM.
  • Human-Level Proximity: Human performance on the same task was measured at 74.02%, indicating that the model is approaching human-level capability in linking informal talk to formal descriptions.
  • Information Filtering: The authors noted that removing "noise" (stopword-heavy long utterances) could improve processing speed without hurting accuracy, suggesting that much of daily conversation is redundant for factual extraction.

Conclusion and Future Work

The study demonstrates that specialized hierarchical attention is crucial for navigating the complexities of multiparty dialog. While the current model excels at identifying character entities, future work will expand to other entity types like locations and objects, and integrate automatic entity linking to handle nicknames and aliases more effectively.

Summary updated

BibTeX
@inproceedings{ma2018challenging,
  author = {Kaixin Ma and Tomasz Jurczyk and Jinho D. Choi},
  title = {{Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Syb3OmWuZS},
}
Figure