Abstract
Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and self-matching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both state-of-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.
Introduction: The Structural Gap in Machine Comprehension
In the field of Machine Comprehension (MC), the goal is to enable models to read a passage and answer questions based on its content. While attention mechanisms have become the standard tool for this task, existing architectures often suffer from two primary limitations. First, there is a lack of structural distinction between different types of attention. Many models treat the task of aligning a question to a passage and the task of understanding internal passage context as a single, iterative process. Second, these models often rely on a single mathematical representation of the question, missing the nuanced "perspectives" that multiple encodings could provide.
Methods: The PhaseCond Architecture
To address these gaps, we introduce Phase Conductor (PhaseCond), a framework that organizes attention into a structured, two-phase pipeline.
Phase 1: Question-Passage (Q-P) Attention
This phase focuses on aligning the passage with the question. It utilizes a stack of $N$ layers to produce question-aware passage representations. To regulate the information flow and prevent overfitting—a common issue when stacking many layers—we employ an "Outer Fusion" layer (implemented as a highway network) at the end of the phase to process the concatenated outputs of all layers.
Phase 2: Evidence Propagation (Self-Attention)
Once the passage has been aligned with the question, Phase 2 uses $K$ layers of self-attention to allow passage words to interact with one another. This step is crucial for resolving coreferences (e.g., linking "JFK" to "Kennedy") and propagating evidence across the text. Between these layers, we use "Inner Fusion" (GRU-like gating) to dynamically decide how much information to retain from previous steps.
Multi-Perspective Encoding
PhaseCond improves the standard dot-product attention by using independent encoders for the question alongside a shared encoder for both the question and passage. This provides the attention function with multiple "perspectives" of the same text, making the resulting representations more robust.
Key Insights and Results
Our evaluation on the SQuAD (Stanford Question Answering Dataset) demonstrates that PhaseCond significantly outperforms competitive single-layered and multi-layered models. A single PhaseCond model achieved an Exact Match (EM) score of 72.1 and an F1 score of 81.4. An ensemble version further pushed these results to 76.1 EM and 84.0 F1, surpassing models like RNET and BiDAF.
Beyond raw metrics, our qualitative analysis revealed fascinating dynamics within the multi-layered structure.
- The "Sharpening" Effect: We observed that in the self-attention phase, the alignment weights in the second layer (Figure 5d) become noticeably "sharper" and more focused than in the first layer (Figure 5c). This suggests that stacking self-attention layers is vital for pinpointing specific evidence within the passage.
- The Risk of Over-Alignment: Interestingly, our experiments showed that adding too many layers to the Q-P phase can be counterproductive. Repeatedly attending to the question can "force" the passage representation to become too similar to the question, causing it to lose the unique contextual information necessary to identify the correct answer boundaries.
Discussion and Limitations
While PhaseCond offers a powerful framework for MC, it comes with increased computational complexity due to the stacking of LSTMs, Highway networks, and GRU gates. Additionally, the current model assumes a fixed sequence (Q-P followed by Self-Attention). Future research could explore whether different tasks benefit from reordering these phases or if the number of layers per phase can be determined dynamically based on the input complexity. Finally, while highly effective on SQuAD, the risk of "over-alignment" may manifest differently on datasets with significantly longer passages or more complex reasoning requirements.
Summary updated