Introduction
Understanding complex narratives requires more than just identifying statistical patterns; it necessitates a grasp of the underlying linguistic structure. This paper introduces DASSA (Discourse-Aware Semantic Self-Attention), a novel approach that integrates semantic roles, coreference chains, and discourse relations directly into the Transformer architecture. By specializing self-attention heads to focus on these structures, the model achieves significant performance gains on narrative reading comprehension tasks, specifically improving the NarrativeQA benchmark by up to +3.4 Rouge-L.
The Challenge of Narrative Understanding
Standard self-attention mechanisms in models like BERT or QANet are "flat," meaning every token can potentially attend to every other token. While powerful, this lack of inductive bias often causes models to struggle with the "connective tissue" of a story—the long-distance dependencies and logical links that define a narrative. For instance, understanding who an actor is across multiple sentences (coreference) or identifying that one event caused another (discourse relations) is essential for answering complex "why" or "how" questions.
Proposed Method: DASSA
The authors propose the DASSA encoder, which builds upon the QANet architecture by specializing multi-head self-attention. The process involves three key steps:
- Linguistic Annotation: The text is pre-processed using off-the-shelf tools to extract Semantic Role Labeling (SRL), Coreference Resolution (Coref), and Discourse Relations (DR).
- Specialized Attention Heads: Specific heads within the multi-head attention mechanism are dedicated to these linguistic relations.
- Symmetric Masking: To guide the model, symmetric masks are applied to these specialized heads. For example, an SRL-focused head is constrained to attend only to tokens within the same semantic argument structure. This reduces the search space and provides a strong structural prior.
This approach maintains low computational complexity by using flat token labels and masks rather than representing full graph edges, avoiding common memory bottlenecks.
Experimental Evaluation
The model was evaluated on the NarrativeQA dataset, which consists of questions based on book summaries and movie scripts. Unlike extractive datasets like SQuAD, NarrativeQA requires synthesizing information across long documents (up to 1,150 tokens).
Key Results
The DASSA-enhanced QANet achieved a Rouge-L score of 53.26, a substantial improvement over the baseline QANet's 48.66. The model also outperformed other strong architectures such as BiAtt+MRU and DecaProp.
Analysis reveals that different linguistic features contribute uniquely to performance:
- SRL is particularly effective for "who" and "when" questions.
- Discourse Relations are crucial for "why" and "where" questions.
- The combination of Explicit Discourse Relations and SRL provided the most consistent overall gains.
Furthermore, the benefits of DASSA scale with context length. In documents exceeding 800 tokens, the structural priors provided by DASSA significantly helped the model navigate long-haul dependencies that typically challenge standard Transformers.
Conclusion and Future Directions
DASSA demonstrates that explicitly baking linguistic structure into self-attention is a powerful way to improve narrative understanding. While the current implementation relies on a pipeline of external annotators—making it sensitive to their errors—it opens the door for more sophisticated knowledge integration. Future work may explore dynamic head allocation across layers or the incorporation of broader world knowledge and commonsense reasoning to further bridge the gap between statistical learning and human-like reading comprehension.
Summary updated