Abstract
Sentence semantic matching (SSM) requires an agent to determine the semantic relation between two sentences, a task foundational to Natural Language Inference (NLI) and Paraphrase Identification (PI). While attention-based and pre-trained language models have advanced this field, many current methods rely on static attention mechanisms that overlook the dynamic nature of human reading. Furthermore, subword tokenization methods like Byte-Pair Encoding (BPE) can result in fragile representations that ignore local phrase structures. We propose the Dynamic Re-read Network (DRr-Net) and its extension, the Locally-Aware Dynamic Re-read Attention Net (LadRa-Net). These models employ a dynamic re-read attention mechanism that iteratively focuses on specific sentence regions. By integrating local structure encoding via Convolutional Neural Networks, LadRa-Net alleviates the shortcomings of BPE and achieves state-of-the-art performance across multiple benchmarks.
Introduction
In Sentence Semantic Matching, the goal is to identify if two sentences, $s^a$ and $s^b$, share a specific relationship (e.g., entailment, contradiction, or equivalence). Despite the success of models like BERT, two primary limitations persist. First, Static Attention mechanisms typically process all "important" words in a single pass, whereas humans naturally shift their focal point as they build a deeper understanding of a text. Second, Subword Fragility caused by BPE can lead to incomplete modeling of local context, such as failing to distinguish between individual words and multi-word entities.
Methodology
Dynamic Re-read Network (DRr-Net)
The core of our approach is the Dynamic Re-read (DRr) Unit. Unlike standard attention, the DRr Unit selects one critical region at each reading step and updates the sentence representation iteratively. This allows the model to "re-read" the most salient parts of the input to refine its semantic understanding.
Locally-Aware Extension: LadRa-Net
To address the limitations of subword encoding, we developed LadRa-Net. This architecture introduces two key components:
- Phrase-CNN (PCNN): This unit uses convolutional layers with varying kernel sizes (e.g., bigrams and trigrams) to capture local structures, ensuring that the model maintains the context of word groupings.
- Dynamic Sequential Attention (DSA): The DSA unit integrates global sentence encodings from pre-trained models (like BERT) with the local features from the PCNN. It utilizes a coverage mechanism to prevent the model from redundantly focusing on the same words while ensuring no critical information is ignored.
Experimental Results
Performance Benchmarks
LadRa-Net was evaluated on six prominent datasets, including SNLI, SciTail, and Quora Question Pairs. The results demonstrate that the dynamic re-read mechanism significantly boosts accuracy:
- SNLI: LadRa-Net (RoBERTa-base) achieved 91.2% accuracy, surpassing the standard RoBERTa-base baseline of 90.9%.
- SciTail: The model reached 93.9% accuracy, a substantial improvement over the DRr-Net baseline (87.4%), highlighting the importance of local structure in complex reasoning tasks.
- Quora: On paraphrase identification, LadRa-Net-BERT reached 92.1%, outperforming base BERT (91.0%).
Psychological Grounding
An interesting finding in our experiments is the optimal "re-read length" ($T$). We found that the model performs best when $T$ is set between 5 and 7 steps. This aligns with psychological research suggesting that human short-term memory and attention can track approximately $7 \pm 2$ objects simultaneously during intensive cognitive tasks.
Conclusion
LadRa-Net effectively mimics human reading behavior through its dynamic re-read attention mechanism. By explicitly modeling local phrase structures, it overcomes the inherent fragility of subword-based pre-trained models. Future work will explore the integration of external knowledge graphs to further enhance the model's reasoning capabilities in complex semantic matching scenarios.
Summary updated