PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge

Yun He, Zhuoer Wang, Yin Zhang, Ruihong Huang, James Caverlee

Abstract

We present PARADE, a new benchmark for paraphrase identification (PI) that necessitates specialized domain knowledge. Unlike traditional datasets where lexical overlap often correlates with semantic equivalence, PARADE contains pairs that are semantically identical despite low lexical similarity, and vice versa. Our experiments demonstrate that state-of-the-art models like BERT and non-expert humans struggle with this task, highlighting a significant gap in current NLP's ability to handle technical reasoning. We make our data and code freely available to support research into knowledge-infused models.

Introduction: Beyond Surface-Level Matching

Paraphrase Identification (PI) is a fundamental task in Natural Language Processing (NLP), determining whether two sentences convey the same meaning. While models perform exceptionally well on general-domain datasets like MRPC (news) and QQP (community forums), they often rely on "surface-level" cues such as lexical overlap (shared words) and syntactic structure.

In specialized fields like Computer Science (CS), these cues are often misleading. True understanding requires domain knowledge to recognize that two linguistically distinct sentences describe the same technical concept. For example, identifying a paraphrase for "Type Inference" requires knowing that specific programming languages like Haskell or Java handle type declarations differently. Existing models excel at pattern matching but struggle with this type of conceptual reasoning.

The PARADE Dataset

To address the limitations of existing benchmarks, we introduced PARADE. The dataset was constructed using a novel pipeline designed to capture technical diversity:

  1. Terminology Extraction: 3,800 CS terms were extracted from thousands of research papers.
  2. Flashcard Mining: These terms were used to query Quizlet, collecting over 30,000 unique entity-definition pairs. This leverages the natural variation in how different individuals define the same technical entity.
  3. Clustering and Annotation: To ensure diversity and avoid redundancy, we used K-means clustering on word embeddings. The final pairs were annotated by 40 CS graduate students (Masters and PhDs) to ensure high-quality labels based on expert knowledge.

The resulting dataset contains 10,182 sentence pairs across 788 CS entities, balanced between paraphrases (46.9%) and non-paraphrases (53.1%).

The Lexical Overlap Trap

A core challenge of PARADE is the decoupling of lexical similarity from semantic equivalence. Traditional models often "cheat" by matching words. However, in PARADE, many paraphrases share very few words, while many non-paraphrases appear deceptively similar due to shared technical terminology that does not imply equivalence.

Distribution of Jaccard values for paraphrase and non-paraphrase pairs in PARADE.

As shown in the distribution of Jaccard values, a significant portion of paraphrases have low lexical overlap, making them difficult for standard pattern-matching models to identify. Conversely, non-paraphrases often exhibit higher overlap than expected, creating a "trap" for models relying on word-matching heuristics.

Experimental Results

We evaluated several state-of-the-art neural models on PARADE. The results reveal a stark performance drop compared to general-domain benchmarks.

Performance comparison of BERT-large across various paraphrase identification datasets.

As seen in the table above, BERT-large, which achieves an F1 score near 0.90 on MRPC and QQP, drops to 0.709 on PARADE. Interestingly, non-expert humans (students not majoring in CS) also performed poorly, achieving an F1 of only 0.56. This confirms that the challenge is rooted in a lack of specialized "world knowledge" rather than just linguistic complexity.

While models like ALBERT-xxlarge (F1: 0.741) and SciBERT (Accuracy: 0.741) show marginal improvements, they still fail to reach the high performance seen in general domains. Qualitative analysis suggests models struggle most with:

  • Specialized Terminology: Failing to link related concepts (e.g., "haskell" vs "java").
  • Acronyms: Missing links like "OS" to "Operating System."
  • Mathematical Inference: Misinterpreting code snippets or equations (e.g., "arr[i][j] == 1").

Conclusion and Future Work

PARADE serves as a rigorous testbed for domain-specific reasoning in NLP. Our findings suggest that simply increasing model size or general pre-training is insufficient for mastering technical domains. Future research should focus on methods for explicitly infusing external knowledge—such as knowledge graphs or technical documentation—into language models to bridge the gap between linguistic pattern matching and true conceptual understanding.

Summary updated

Abstract

We present a new benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. PARADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non-paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annotators have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that incorporate domain knowledge. We make our data and code freely available.
BibTeX
@inproceedings{he2020parade,
  author = {Yun He and Zhuoer Wang and Yin Zhang and Ruihong Huang and James Caverlee},
  title = {{PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.611},
}
Figure