Automatic Biomedical Term Clustering by Learning Fine-grained Term Representations

Sihang Zeng, Zheng Yuan, Sheng Yu

Abstract

Building comprehensive Biomedical Knowledge Graphs (BKGs) requires the accurate clustering of diverse textual strings into unified semantic concepts. While existing representation models excel at identifying synonyms, they often struggle with "near-miss" cases—terms that are textually similar but semantically distinct, such as "Type 1 Diabetes" and "Type 2 Diabetes." This paper introduces CODER++, a model that utilizes dynamic hard negative sampling to learn fine-grained term representations. By iteratively identifying and distinguishing confusing term pairs during training, CODER++ achieves a significant performance leap, improving the F1-score on UMLS clustering benchmarks from 0.28 to 0.64.

Introduction

In the biomedical domain, the same concept can be expressed through numerous surface forms (e.g., "heart attack" vs. "myocardial infarction"). Term clustering is the process of grouping these strings into a single concept, a foundational step for constructing Biomedical Knowledge Graphs. Current state-of-the-art models like SapBERT and the original CODER utilize contrastive learning to align synonymous terms. However, these models frequently suffer from a "collapsing" problem: because they are trained against random negative samples, they fail to distinguish between terms that share significant character overlap but represent different medical conditions. This lack of granularity leads to clinical inaccuracies when distinct concepts are merged during clustering.

Methodology

CODER++ addresses the limitations of previous models through a refined contrastive learning framework focused on fine-grained discrimination. The core innovations include:

1. Dynamic Hard Negative Sampling

Standard contrastive learning often relies on random terms within a mini-batch as negative examples. For a term like "Type 1 Diabetes," a random negative might be "Lung Cancer," which is easily distinguishable. CODER++ instead employs a Faiss index to perform a similarity search across the entire dataset, identifying "hard negatives"—terms the model currently perceives as similar but which belong to different concepts.

2. Iterative Index Updates

As the model's embeddings evolve during training, the set of terms that are "confusing" also changes. To prevent overfitting to a static set of hard negatives, the authors update the Faiss index every 60,000 steps (approximately once per epoch). This dynamic refresh ensures the model is consistently challenged by the most difficult examples throughout the training process.

3. Multi-Similarity (MS) Loss

The training process utilizes Multi-Similarity loss to guide contrastive learning. This loss function allows the model to weigh the importance of different pairs within a mini-batch, focusing more heavily on the most informative (i.e., the most confusing) samples.

4. Scalable Evaluation

Evaluating clustering across the 10 million+ terms in the Unified Medical Language System (UMLS) is computationally prohibitive with standard $O(n^2)$ algorithms. The authors developed a specialized evaluation algorithm that reduces complexity to $O(n)$, enabling feasible large-scale benchmarking.

Experimental Setup and Datasets

The model was pre-trained and evaluated using the UMLS 2020 AA release. To ensure the model remained a versatile tool for general biomedical NLP, it was also tested on BC5CDR for zero-shot term normalization (linking terms to a fixed dictionary). Finally, the model was applied to BIOS, a newly released Biomedical Knowledge Graph, to demonstrate its real-world utility in concept clustering.

Key Insights

  • The "Easy Negative" Trap: The research demonstrates that standard contrastive learning is often "too easy." Without hard negatives, models never learn to attend to the fine-grained modifiers (like "Type 1" vs. "Type 2") that define medical specificity.
  • Clustering vs. Linking: The authors highlight that clustering is inherently more difficult than entity linking. While linking benefits from a constrained search space (matching to a known ID), clustering requires the model to determine if a term belongs to an existing group or represents an entirely new concept, necessitating much higher precision.
  • The Necessity of Dynamics: Ablation studies confirmed that static hard negative sampling leads to poor generalization. Only by dynamically updating the negative samples as the model improves can robust, fine-grained representations be achieved.

Results

CODER++ demonstrates a substantial improvement over previous benchmarks in both precision and overall F1-score:

  • Clustering Performance: On the UMLS benchmark, CODER++ achieved an F1-score of 0.644, a massive increase over SapBERT (0.284) and the original CODER (0.121).
  • Precision and Specificity: The model reached a precision of 0.732, more than double that of SapBERT (0.302). This indicates a significantly lower rate of "false merges" between distinct concepts.
  • Qualitative Analysis: In specific case studies, CODER++ assigned a similarity score of only 0.502 to "Type 1 Diabetes" vs. "Type 2 Diabetes." In contrast, previous models assigned scores exceeding 0.90, effectively treating them as synonyms.
  • Generalization: Despite its focus on clustering, CODER++ maintained or slightly improved performance on standard zero-shot normalization tasks (BC5CDR), proving that fine-grained training does not degrade general semantic understanding.

Conclusion and Future Work

CODER++ represents a significant advancement in biomedical term representation by solving the "near-miss" problem through dynamic hard negative sampling. While the model is computationally intensive—requiring 9 days of training on 8 NVIDIA A100 GPUs—the resulting gains in clustering accuracy are vital for the integrity of Biomedical Knowledge Graphs. Future research may explore extending the model to handle longer, more complex medical descriptions beyond the 32-token limit used in this study, and incorporating more diverse relational information beyond synonymy.

Summary updated

BibTeX
@inproceedings{zeng2022automatic,
  author = {Sihang Zeng and Zheng Yuan and Sheng Yu},
  title = {{Automatic Biomedical Term Clustering by Learning Fine-grained Term Representations}},
  booktitle = {BioNLP @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=GMxUm4nz0o},
}
Figure