Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension

Todor Mihaylov, Anette Frank

Introduction

Understanding complex narratives requires more than just identifying statistical patterns; it necessitates a grasp of the underlying linguistic structure. This paper introduces DASSA (Discourse-Aware Semantic Self-Attention), a novel approach that integrates semantic roles, coreference chains, and discourse relations directly into the Transformer architecture. By specializing self-attention heads to focus on these structures, the model achieves significant performance gains on narrative reading comprehension tasks, specifically improving the NarrativeQA benchmark by up to +3.4 Rouge-L.

The Challenge of Narrative Understanding

Standard self-attention mechanisms in models like BERT or QANet are "flat," meaning every token can potentially attend to every other token. While powerful, this lack of inductive bias often causes models to struggle with the "connective tissue" of a story—the long-distance dependencies and logical links that define a narrative. For instance, understanding who an actor is across multiple sentences (coreference) or identifying that one event caused another (discourse relations) is essential for answering complex "why" or "how" questions.

Proposed Method: DASSA

The authors propose the DASSA encoder, which builds upon the QANet architecture by specializing multi-head self-attention. The process involves three key steps:

  1. Linguistic Annotation: The text is pre-processed using off-the-shelf tools to extract Semantic Role Labeling (SRL), Coreference Resolution (Coref), and Discourse Relations (DR).
  2. Specialized Attention Heads: Specific heads within the multi-head attention mechanism are dedicated to these linguistic relations.
  3. Symmetric Masking: To guide the model, symmetric masks are applied to these specialized heads. For example, an SRL-focused head is constrained to attend only to tokens within the same semantic argument structure. This reduces the search space and provides a strong structural prior.
Overview of the DASSA architecture, showing the integration of linguistic masks into the multi-head self-attention mechanism.

This approach maintains low computational complexity by using flat token labels and masks rather than representing full graph edges, avoiding common memory bottlenecks.

Experimental Evaluation

The model was evaluated on the NarrativeQA dataset, which consists of questions based on book summaries and movie scripts. Unlike extractive datasets like SQuAD, NarrativeQA requires synthesizing information across long documents (up to 1,150 tokens).

Key Results

The DASSA-enhanced QANet achieved a Rouge-L score of 53.26, a substantial improvement over the baseline QANet's 48.66. The model also outperformed other strong architectures such as BiAtt+MRU and DecaProp.

Analysis reveals that different linguistic features contribute uniquely to performance:

  • SRL is particularly effective for "who" and "when" questions.
  • Discourse Relations are crucial for "why" and "where" questions.
  • The combination of Explicit Discourse Relations and SRL provided the most consistent overall gains.
Performance improvement of DASSA variants over the QANet baseline across different question types.

Furthermore, the benefits of DASSA scale with context length. In documents exceeding 800 tokens, the structural priors provided by DASSA significantly helped the model navigate long-haul dependencies that typically challenge standard Transformers.

Conclusion and Future Directions

DASSA demonstrates that explicitly baking linguistic structure into self-attention is a powerful way to improve narrative understanding. While the current implementation relies on a pipeline of external annotators—making it sensitive to their errors—it opens the door for more sophisticated knowledge integration. Future work may explore dynamic head allocation across layers or the incorporation of broader world knowledge and commonsense reasoning to further bridge the gap between statistical learning and human-like reading comprehension.

Summary updated

Abstract

In this work, we propose to use linguistic annotations as a basis for a \textit{Discourse-Aware Semantic Self-Attention} encoder that we employ for reading comprehension on long narrative texts. We extract relations between discourse units, events and their arguments as well as coreferring mentions, using available annotation tools. Our empirical evaluation shows that the investigated structures improve the overall performance, especially intra-sentential and cross-sentential discourse relations, sentence-internal semantic role relations, and long-distance coreference relations. We show that dedicating self-attention heads to intra-sentential relations and relations connecting neighboring sentences is beneficial for finding answers to questions in longer contexts. Our findings encourage the use of discourse-semantic annotations to enhance the generalization capacity of self-attention models for reading comprehension.
BibTeX
@inproceedings{mihaylov2019discourse,
  author = {Todor Mihaylov and Anette Frank},
  title = {{Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1257},
}
Figure