LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching

Kun Zhang, Guangyi Lv, Le Wu, Enhong Chen, Qi Liu, Meng Wang

Abstract

Sentence semantic matching (SSM) requires an agent to determine the semantic relation between two sentences, a task foundational to Natural Language Inference (NLI) and Paraphrase Identification (PI). While attention-based and pre-trained language models have advanced this field, many current methods rely on static attention mechanisms that overlook the dynamic nature of human reading. Furthermore, subword tokenization methods like Byte-Pair Encoding (BPE) can result in fragile representations that ignore local phrase structures. We propose the Dynamic Re-read Network (DRr-Net) and its extension, the Locally-Aware Dynamic Re-read Attention Net (LadRa-Net). These models employ a dynamic re-read attention mechanism that iteratively focuses on specific sentence regions. By integrating local structure encoding via Convolutional Neural Networks, LadRa-Net alleviates the shortcomings of BPE and achieves state-of-the-art performance across multiple benchmarks.

Introduction

In Sentence Semantic Matching, the goal is to identify if two sentences, $s^a$ and $s^b$, share a specific relationship (e.g., entailment, contradiction, or equivalence). Despite the success of models like BERT, two primary limitations persist. First, Static Attention mechanisms typically process all "important" words in a single pass, whereas humans naturally shift their focal point as they build a deeper understanding of a text. Second, Subword Fragility caused by BPE can lead to incomplete modeling of local context, such as failing to distinguish between individual words and multi-word entities.

Methodology

Dynamic Re-read Network (DRr-Net)

The core of our approach is the Dynamic Re-read (DRr) Unit. Unlike standard attention, the DRr Unit selects one critical region at each reading step and updates the sentence representation iteratively. This allows the model to "re-read" the most salient parts of the input to refine its semantic understanding.

The Architecture of Dynamic Re-read Network (DRr-Net) and its internal units.

Locally-Aware Extension: LadRa-Net

To address the limitations of subword encoding, we developed LadRa-Net. This architecture introduces two key components:

  1. Phrase-CNN (PCNN): This unit uses convolutional layers with varying kernel sizes (e.g., bigrams and trigrams) to capture local structures, ensuring that the model maintains the context of word groupings.
  2. Dynamic Sequential Attention (DSA): The DSA unit integrates global sentence encodings from pre-trained models (like BERT) with the local features from the PCNN. It utilizes a coverage mechanism to prevent the model from redundantly focusing on the same words while ensuring no critical information is ignored.
The full architecture of LadRa-Net, incorporating Phrase-CNN (PCNN) and Dynamic Sequential Attention (DSA).

Experimental Results

Performance Benchmarks

LadRa-Net was evaluated on six prominent datasets, including SNLI, SciTail, and Quora Question Pairs. The results demonstrate that the dynamic re-read mechanism significantly boosts accuracy:

  • SNLI: LadRa-Net (RoBERTa-base) achieved 91.2% accuracy, surpassing the standard RoBERTa-base baseline of 90.9%.
  • SciTail: The model reached 93.9% accuracy, a substantial improvement over the DRr-Net baseline (87.4%), highlighting the importance of local structure in complex reasoning tasks.
  • Quora: On paraphrase identification, LadRa-Net-BERT reached 92.1%, outperforming base BERT (91.0%).

Psychological Grounding

An interesting finding in our experiments is the optimal "re-read length" ($T$). We found that the model performs best when $T$ is set between 5 and 7 steps. This aligns with psychological research suggesting that human short-term memory and attention can track approximately $7 \pm 2$ objects simultaneously during intensive cognitive tasks.

Conclusion

LadRa-Net effectively mimics human reading behavior through its dynamic re-read attention mechanism. By explicitly modeling local phrase structures, it overcomes the inherent fragility of subword-based pre-trained models. Future work will explore the integration of external knowledge graphs to further enhance the model's reasoning capabilities in complex semantic matching scenarios.

Summary updated

Abstract

Sentence semantic matching requires an agent to determine the semantic relation between two sentences, which is widely used in various natural language tasks, such as Natural Language Inference (NLI), Paraphrase Identification (PI), and so on. Much recent progress has been made in this area, especially attention-based methods and pre-trained language model based methods. However, most of these methods focus on all the important parts in sentences in a static way and only emphasize how important the words are to the query, inhibiting the ability of attention mechanism. In order to overcome this problem and boost the performance of attention mechanism, we propose a novel dynamic re-read attention, which can pay close attention to one small region of sentences at each step and re-read the important parts for better sentence representations. Based on this attention variation, we develop a novel Dynamic Re-read Network (DRr-Net) for sentence semantic matching. Moreover, selecting one small region in dynamic re-read attention seems insufficient for sentence semantics, and employing pre-trained language models as input encoders will introduce incomplete and fragile representation problems. To this end, we extend DRrNet to Locally-Aware Dynamic Re-read Attention Net (LadRa-Net), in which local structure of sentences is employed to alleviate the shortcoming of Byte-Pair Encoding (BPE) in pre-trained language models and boost the performance of dynamic reread attention. Extensive experiments on two popular sentence semantic matching tasks demonstrate that DRr-Net can significantly improve the performance of sentence semantic matching. Meanwhile, LadRa-Net is able to achieve better performance by considering the local structures of sentences. In addition, it is exceedingly interesting that some discoveries in our experiments are consistent with some findings of psychological research.
BibTeX
@article{zhang2023ladra,
  author = {Kun Zhang and Guangyi Lv and Le Wu and Enhong Chen and Qi Liu and Meng Wang},
  title = {{LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  doi = {10.1109/tnnls.2021.3103185},
}
Figure