Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching

Bo Shao, Yeyun Gong, Weizhen Qi, Nan Duan, Xiaola LinEMNLP 2019 · DOI

Abstract

Standard applications of Bidirectional Encoder Representations from Transformers (BERT) for classification tasks typically rely on the representation of a single special token ([CLS]). This paper introduces a hybrid architecture that combines BERT's contextual power with the fine-grained interaction capabilities of MatchLSTM. By moving beyond the "one-vector-fits-all" approach, the authors capture word-by-word interactions between sentence pairs. The proposed BERT+MatchLSTM model achieves state-of-the-art results on the WikiQA and SNLI benchmarks, demonstrating the importance of granular aggregation in sequence matching tasks.

Introduction

Sequence matching is a fundamental task in Natural Language Processing (NLP), involving the determination of relationships between text snippets, such as question-answering or natural language inference. While BERT has significantly advanced the state-of-the-art, its reliance on the [CLS] token for classification often ignores rich, word-level interactions. This study addresses the limitation of BERT's standard aggregation method by proposing a more sophisticated interaction layer to capture deeper relational nuances.

Methodology

The proposed architecture stacks a MatchLSTM layer on top of a pre-trained BERT-Base encoder. The process follows three distinct stages:

  1. BERT Encoding: A sentence pair is processed by BERT. Instead of extracting only the [CLS] token, the model retrieves the final hidden states for every word in both sequences.
  2. MatchLSTM Interaction: The BERT outputs are split into two sequences. A MatchLSTM layer performs word-by-word attention, calculating weighted scores for every word in the second sentence against all words in the first. This explicitly matches related terms across the sequences.
  3. Aggregation & Classification: The final hidden state of the MatchLSTM is concatenated with BERT’s original [CLS] token (referred to as the First Hidden State or FHS). This combined vector is then passed through a softmax layer for final prediction.

The authors also evaluated alternative aggregation methods, including simple averaging and Convolutional Neural Networks (CNNs), to establish the superiority of the MatchLSTM approach.

Experimental Setup

The model was evaluated on two primary benchmarks:

  • WikiQA: A sentence-level dataset for Answer Sentence Selection, requiring the identification of correct answers to queries from Wikipedia snippets.
  • SNLI (Stanford Natural Language Inference): A large-scale dataset of 570,000 sentence pairs categorized by entailment, contradiction, or neutrality.

Results and Analysis

The BERT+MatchLSTM model consistently outperformed vanilla BERT and other competitive baselines across both datasets.

WikiQA Performance

On the WikiQA dataset, the proposed model achieved a Mean Average Precision (MAP) of 83.53 and a Mean Reciprocal Rank (MRR) of 85.13. This represents a significant improvement over the standard BERT-Base, which scored 80.15 MAP and 82.31 MRR.

Performance comparison on the WikiQA dataset.

SNLI Performance

For the SNLI benchmark, the model reached a test accuracy of 90.9%. This surpasses vanilla BERT-Base (89.6%) and other high-performing models like DRCN+Ensemble (90.1%).

Performance comparison on the SNLI dataset.

Key Insights

  • Granularity and Alignment: Explicit word-to-word alignment via MatchLSTM is crucial for tasks where specific keywords must align between sequences.
  • The Importance of FHS: Retaining the [CLS] token (FHS) in the final feature vector was vital; removing it (the "-FHS" variant) led to a significant performance drop.
  • Superiority of MatchLSTM: MatchLSTM outperformed CNN and Average pooling, suggesting that relational context and word order are essential for effective sequence matching.

Conclusion

The integration of MatchLSTM with BERT provides a powerful mechanism for capturing fine-grained interactions in sequence matching. While the addition of an LSTM layer increases computational overhead and inference time, the gains in accuracy on WikiQA and SNLI are substantial. Future work may explore the application of this aggregation method to larger BERT variants and a broader range of NLP tasks.

Summary updated

BibTeX
@inproceedings{shao2019aggregating,
  author = {Bo Shao and Yeyun Gong and Weizhen Qi and Nan Duan and Xiaola Lin},
  title = {{Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1626},
}
Figure