Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension

Todor Mihaylov, Anette Frank

Introduction

Transformer-based models have become the standard for natural language processing, yet they often struggle with narrative reading comprehension due to their "linguistically agnostic" nature. These models treat text as a sequence of tokens and must learn structural relationships—such as event chains and character movements—entirely from scratch. To bridge this gap, the authors introduce DASSA (Discourse-Aware Semantic Self-Attention), a method that explicitly bakes linguistic structure into the Transformer's self-attention mechanism.

Methodology: Integrating Linguistic Structure

The DASSA encoder builds upon the high-performance QANet architecture. Its core innovation lies in "specializing" certain self-attention heads to focus on specific linguistic relations using pre-computed annotations:

  • Semantic Role Labeling (SRL): Heads focus on the relationship between predicates (verbs) and their arguments (who, what, where).
  • Coreference Resolution (Coref): Heads track different mentions of the same entity, such as linking a name to a pronoun.
  • Shallow Discourse Parsing (DR): Heads analyze how sentences or clauses relate to one another (e.g., causal or temporal links).
The DASSA architecture (B) modifies standard self-attention (A) by using attention masks (D) derived from linguistic annotations to guide specific heads (C).

To implement this efficiently, the authors use attention masks. For a specialized head, the mask restricts attention to tokens within the same semantic or discourse frame. This approach leverages structured knowledge while maintaining the parallelizable nature and low computational overhead of the original Transformer.

Evaluation and Key Insights

The model was evaluated on the NarrativeQA benchmark, which involves answering questions based on long summaries of books and movie scripts. Unlike short-context datasets, NarrativeQA requires synthesizing information across contexts of up to 1,150 tokens.

Performance and Specialization

DASSA-enhanced QANet achieved an improvement of up to +3.4 Rouge-L points over the baseline. The study revealed that not all linguistic information is equally beneficial; Explicit Discourse Relations and SRL provided the most significant performance boosts.

Performance improvement over the QANet baseline across different question types and linguistic annotation combinations.

Furthermore, the benefits of DASSA are highly specific to the query type. For example, SRL is a powerful tool for "Who" and "When" questions, whereas Discourse Relations are vital for answering "Why" and "Where" questions. The structural guidance also proves more valuable as context length increases, helping the model ignore irrelevant noise in 800-1000 token passages.

Conclusion

By providing a linguistic "skeleton" for the model to follow, DASSA significantly enhances narrative understanding in long-form text. While the current model depends on external annotation tools, future research may move toward joint learning frameworks. Such systems would learn to perform linguistic tasks and reading comprehension simultaneously, allowing for more dynamic and robust information processing.

Summary updated

BibTeX
@inproceedings{mihaylov2019discourse,
  author = {Todor Mihaylov and Anette Frank},
  title = {{Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1257},
}
Figure