Knowledge-Guided Paraphrase Identification

Haoyu Wang, Fenglong Ma, Yaqing Wang, Jing Gao

Abstract

Paraphrase identification (PI) in specialized domains like medicine or computer science is challenging due to the lack of domain-specific context in general-purpose models. We present Knowing, a framework that enhances BERT-based PI by retrieving and fusing external knowledge from Wikipedia outlines. Using a gated mechanism to balance internal semantics with external information, Knowing achieves significant performance gains on domain-specific benchmarks like PARADE and clinicalSTS2019, outperforming both general and domain-specialized pre-trained models.

Introduction

Paraphrase Identification (PI) is the task of determining if two sentences share the same meaning. While large language models like BERT excel at this in general contexts, they often stumble in specialized domains where technical jargon requires external context. For instance, a general model might fail to recognize that "the lowest level of code made up of 0s and 1s" is a paraphrase of "binary instructions used by the cpu." Without domain-specific knowledge, models rely too heavily on surface-level lexical overlap.

The authors identify two primary challenges in solving this problem:

  1. Knowledge Selection: Efficiently identifying relevant information within massive domain-specific corpora.
  2. Knowledge Fusion: Integrating unstructured external information into neural architectures without introducing performance-degrading noise.

The Knowing Framework

The proposed Knowing framework addresses these challenges through a modular architecture consisting of four components:

  1. Base Prediction Module: Utilizes a standard BERT encoder to capture lexical and syntactic features of the sentence pair.
  2. Knowledge Selection Module: Employs the BM25 algorithm to retrieve the top-$m$ relevant Wikipedia "outlines"—concise lead sections of articles—avoiding the computational overhead of dense retrieval.
  3. Knowledge Fusion Module: Encodes retrieved outlines via BERT and applies an attention mechanism to weight them based on their relevance to the input sentences, synthesizing a single knowledge representation.
  4. Aggregation Module: Integrates the base prediction and knowledge representation using a gated mechanism. This gate uses BM25 relevance scores to dynamically decide the influence of external knowledge on the final prediction.
The Knowing framework architecture, featuring modules for base prediction, knowledge selection, fusion, and gated aggregation.

Experimental Evaluation

The model was evaluated on two domain-specific benchmarks: PARADE (computer science) and clinicalSTS2019 (biomedical/clinical). The authors converted the clinicalSTS2019 similarity scores into binary labels to fit the PI task.

Key Results

Knowing consistently outperformed state-of-the-art baselines, including vanilla BERT and ALBERT. On the PARADE dataset, Knowing achieved an F1 score of 0.7338 (a 3.6% improvement over BERT). On clinicalSTS2019, the improvement was more pronounced, reaching an F1 of 0.8714 (a 9.4% improvement).

Performance metrics (Accuracy, F1, Precision, Recall) for the Knowing framework on the PARADE and clinicalSTS2019 datasets.

Discussion and Insights

  • Outlines vs. Full Text: Wikipedia "outlines" proved more effective than full articles, providing high-level abstractions that fit BERT’s token limits while minimizing noise.
  • Retrieval vs. Pre-training: Explicit knowledge retrieval at inference time outperformed domain-specific pre-trained models like SciBERT and BioMedBERT. This "open-book" approach allows for more accurate factual access than relying on static weights learned during pre-training.
  • Gating Mechanism: The gate is critical for robustness. If retrieved knowledge is low-quality (indicated by low BM25 scores), the gate reduces its influence, preventing external noise from degrading performance on simpler pairs.

Conclusion

The Knowing framework demonstrates that external knowledge retrieval is a powerful alternative to specialized pre-training for domain-specific PI. Future work may explore incorporating structured data, such as Knowledge Graphs, to provide even more precise relational context. Current limitations include a dependency on Wikipedia coverage and increased computational overhead during inference.

Summary updated

BibTeX
@inproceedings{wang2021knowledge,
  author = {Haoyu Wang and Fenglong Ma and Yaqing Wang and Jing Gao},
  title = {{Knowledge-Guided Paraphrase Identification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.72},
}
Figure