Can Embeddings Adequately Represent Medical Terminology? New Large-Scale Medical Term Similarity Datasets Have the Answer!

Claudia Schulz, Damir JuricAAAI 2020 · DOI

Introduction

In the general Natural Language Processing (NLP) domain, massive benchmarks are used to evaluate whether models truly understand semantic relationships. In medicine, however, researchers have long been "flying blind," relying on tiny datasets like MiniMayoSRS (29 pairs) to evaluate how well embeddings capture medical meaning. This lack of scale is problematic because medical terminology is vast and complex, often featuring multi-word terms that are "lexically deceptive"—appearing visually similar but meaning entirely different things.

Researchers from Babylon Health have addressed this gap by introducing the first large-scale benchmarks for medical term similarity, moving from datasets of a few hundred pairs to over 600,000. Their findings serve as a wake-up call: while current AI models perform well on simple datasets, they struggle significantly with the complex terminology used by clinicians, often confusing lexical similarity with actual medical meaning.

Methodology: Harnessing SNOMED CT

The authors leveraged SNOMED CT, the world’s most comprehensive clinical terminology system, to automatically generate massive similarity datasets. Their approach involved two key innovations:

  1. Positive Instance Extraction: Using SNOMED’s internal structure (synonyms and deactivated concepts) to find semantically identical pairs.
  2. Hard Negative Sampling: Using Levenshtein distance (edit distance) to identify terms that look similar but are medically unrelated. This creates a "hard" dataset that tests whether a model is simply matching characters or actually understanding medical meaning.

The New Large-Scale Benchmarks

The resulting datasets, such as FSN-SYN and SYN-SYN, represent a massive leap in scale compared to previous benchmarks. Notably, 92% of the terms in these new datasets are multi-word, reflecting real-world clinical documentation far better than previous benchmarks which were almost entirely composed of single-word terms.

Table 2: Statistics of the new large-scale medical similarity datasets, showing the breakdown of easy and hard pairs across different relationship types.

Key Results: The "Lexical Trap"

The study evaluated a wide array of models, including traditional word embeddings (GloVe, FastText) and modern contextual models (BERT, SciBERT, ELMo, GPT). The results revealed several critical insights:

  • Small Datasets are Misleading: On small datasets, almost all models appear competent. The new large-scale data reveals massive performance gaps that were previously invisible.
  • The Failure of Contextual Models: Contextual models like BERT and ELMo performed poorly on "hard" negative pairs. They frequently predicted high similarity based on character overlap, even when meanings were unrelated. On some "hard" tests, these models achieved an AUC of less than 0.5, meaning they were worse than random guessing.
  • Knowledge-Augmented Wins: Embeddings trained using the MeSH (Medical Subject Headings) thesaurus performed the best. This suggests that raw text exposure is insufficient; AI requires structured medical knowledge to truly represent terminology.
Table 5: Performance comparison across models on the new benchmarks. Note the significant performance drop on 'hard' datasets for contextual models like BERT and GPT.

Conclusion

The Babylon Health study demonstrates that current AI models often rely on surface-level lexical similarity rather than true medical understanding. The results highlight a pressing need for "knowledge-aware" embeddings that move beyond simple co-occurrence and incorporate the hierarchical logic of medical ontologies. By releasing their code, the authors enable the community to leverage these large-scale benchmarks to develop more robust and medically accurate AI systems.

Summary updated

BibTeX
@inproceedings{schulz2020can,
  author = {Claudia Schulz and Damir Juric},
  title = {{Can Embeddings Adequately Represent Medical Terminology? New Large-Scale Medical Term Similarity Datasets Have the Answer!}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6404},
}

Loading related content...

Figure