CSDR-BERT: a pre-trained scientific dataset match model for Chinese Scientific Dataset Retrieval

Xintao Chu, Jianping Liu, Jian Wang, Yingfei Wang, Meng Wang, Xunxun GuarXiv 2023 · DOI · arXiv

Abstract

As the open science movement expands, the volume of shared scientific datasets has grown exponentially. However, retrieving specific datasets remains difficult due to the "vocabulary mismatch problem," where different scientific fields use varying terminology for identical concepts. This paper presents CSDR-BERT, a specialized retrieval model that enhances the Sentence-BERT architecture with unsupervised contrastive learning (SimCSE), a K-Nearest Neighbors (KNN) matching layer, and the Cosent loss function. Evaluated on a custom Chinese scientific dataset and standard benchmarks, CSDR-BERT demonstrates significant improvements in semantic matching accuracy, achieving an F-score of 92.49% on domain-specific tasks.

Introduction

The effective discovery of scientific datasets is a cornerstone of modern research. Despite the proliferation of data repositories, traditional keyword-based search engines often fail to bridge the gap between user queries and dataset metadata. This failure is primarily attributed to the vocabulary mismatch problem: scientific concepts are frequently represented by synonyms (e.g., "chickenpox" vs. "varicella") or distinct notations (e.g., "glucose" vs. "C6H12O6").

While general-purpose pre-trained language models like BERT have advanced natural language processing, they often lack the domain-specific nuances required for specialized scientific retrieval. CSDR-BERT addresses these limitations by refining sentence representations and leveraging local data similarities to improve retrieval precision.

Methodology

CSDR-BERT builds upon the Sentence-BERT (SBERT) framework, introducing three primary architectural enhancements to better capture scientific semantics:

  1. Unsupervised SimCSE (Contrastive Learning): To produce more robust sentence embeddings, the model employs Simple Contrastive Learning of Sentence Embeddings (SimCSE). By passing the same sentence through the model twice with different dropout masks, the model learns to cluster similar meanings more effectively in the vector space.
  2. KNN-BERT Matching Layer: Beyond standard linear classification, the authors integrated a K-Nearest Neighbors (KNN) component. The final matching score is a weighted combination of the neural network's output and the KNN result, allowing the model to benefit from local similarities in the training data.
  3. Cosent Loss Function: The model utilizes Cosine Sentence (Cosent) loss instead of standard cross-entropy. Cosent focuses on the relative similarity between pairs, ensuring that relevant matches are ranked significantly higher than non-matches, which is critical for retrieval performance.
Figure 1: The architecture of the CSDR-BERT model, illustrating the integration of SimCSE, pooling layers, and the hybrid KNN matching layer.

Datasets and Experimental Setup

The model was trained and evaluated using a combination of domain-specific and general-purpose datasets:

  • Scientific Dataset: A custom benchmark derived from the National Earth System Science Data Centre (NESDC), containing over 30,000 headings. A subset of 10,960 pairs was manually annotated for similarity.
  • Chinese Scientific Literature (CSL): A corpus of approximately 396,000 journal articles. A subset of 10,000 articles was used for unsupervised pre-training via SimCSE.
  • Public Benchmarks: To ensure generalizability, the model was tested on standard Chinese NLP tasks, including AFQMC, LCQMC, STS-B, and PAWS-X.

Results and Discussion

Experimental results indicate that CSDR-BERT significantly outperforms baseline models. On the specialized Scientific Dataset, the full model achieved an F-score of 92.49% and an Accuracy of 92.66%, representing a substantial improvement over standard Sentence-BERT configurations.

Ablation studies highlight the contribution of each component. Integrating the KNN layer alone provided a 3.88% boost in accuracy, while the Cosent loss and SimCSE pre-training added 1.45% and 0.49%, respectively.

Table 1: Performance comparison and ablation results on the Scientific Dataset, demonstrating the impact of SimCSE, Cosent loss, and KNN integration.

The model also showed strong cross-domain performance. On the STS-B benchmark, CSDR-BERT reached a Spearman correlation of 80.85%, compared to 70.79% for the baseline. Notably, the model excelled on the PAWS-X dataset, which is designed to challenge models with "word-stacking" (sentences with identical words in different orders), proving its ability to grasp deep semantics rather than just surface-level word overlap.

Conclusion

CSDR-BERT effectively mitigates the vocabulary mismatch problem in Chinese scientific dataset retrieval. By combining contrastive learning with neighborhood-based matching and a ranking-focused loss function, the model provides a robust solution for navigating complex scientific terminology. Future research will focus on incorporating multi-modal metadata—such as temporal and geographical ranges—into the matching process and refining re-ranking algorithms for even higher retrieval precision.

Summary updated

BibTeX
@misc{chu2023csdr,
  author = {Xintao Chu and Jianping Liu and Jian Wang and Yingfei Wang and Meng Wang and Xunxun Gu},
  title = {{CSDR-BERT: a pre-trained scientific dataset match model for Chinese Scientific Dataset Retrieval}},
  year = {2023},
  eprint = {2301.12700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.12700},
}
Figure