Word embeddings trained on published case reports are lightweight, effective for clinical tasks, and free of protected health information

Zachary Flamholz, Lyle Ungar, Gary E. WeissmanmedRxiv 2019 · DOI

Abstract

Rationale: Word embeddings are used to create vector representations of text data but not all embeddings appropriately capture clinical information, are free of protected health information, and are computationally accessible to most researchers. Methods: We trained word embeddings on published case reports because their language mimics that of clinical notes, the manuscripts are already de-identified by virtue of being published, and the corpus is much smaller than those trained on large, publicly available datasets. We tested the performance of these embeddings across five clinically relevant tasks and compared the results to embeddings trained on a large Wikipedia corpus, all publicly available manuscripts, notes from the MIMIC-III database using fastText, GloVe, and word2vec, and using different dimensions. Tasks included clinical applications of lexicographic coverage, semantic similarity, clustering purity, linguistic regularity, and mortality prediction. Results: The embeddings trained using the published case reports performed as well as if not better on most tasks than those using other corpora. The embeddings trained using all published manuscripts had the most consistent performance across all tasks and required a corpus with 100 times as many tokens as the corpus comprised of only case reports. Embeddings trained on the MIMIC-III dataset had small but marginally better scores on the clustering tasks which was also based on clinical notes from the MIMIC-III dataset. Embeddings trained on the Wikipedia corpus, although containing almost twice as many tokens as all available published manuscripts, performed poorly compared to those trained on medical and clinical corpora. Conclusion: Word embeddings trained on freely available published case reports performed well for most clinical task, are free of protected health information, and are small compared to commonly used embeddings trained on larger clinical and non-clinical corpora. The optimal corpus, dimension size, and which embedding model to use for a given task involves tradeoffs in privacy, reproducibility, performance, and computational resources.

Introduction

In Natural Language Processing (NLP), word embeddings serve as the mathematical foundation for representing linguistic relationships. However, clinical text presents a unique challenge due to its specialized jargon and abbreviations. Researchers typically face a "trilemma" when developing clinical models: hospital-sourced data (like MIMIC-III) is high-performing but restricted by privacy laws (PHI); general corpora (like Wikipedia) are shareable but lack medical nuance; and massive biomedical databases (like PubMed) require significant computational resources.

This study explores a "Goldilocks" solution: published medical case reports. These narratives are publicly available, already de-identified, and written in a style that closely mimics actual clinical notes, offering a lightweight yet effective proxy for private hospital data.

Methodology

The research team compared three popular embedding architectures—word2vec, fastText, and GloVe—across four distinct corpora:

  • OA-CR (Open Access Case Reports): A focused subset of PubMed (approx. 50 million tokens).
  • OA-All: All available open-access manuscripts from PubMed Central (approx. 1.8 billion tokens).
  • MIMIC-III: Restricted-access de-identified ICU notes (approx. 148 million tokens).
  • Wikipedia: A general knowledge English corpus (approx. 2.5 billion tokens).

Models were evaluated on tasks including lexicographic coverage, semantic similarity, clustering purity, linguistic regularity, and mortality prediction.

Results: Quality Over Quantity

The most striking finding was that the small OA-CR dataset performed on par with, and often better than, the massive Wikipedia corpus. This suggests that for clinical NLP, the relevance of the prose is more critical than the sheer volume of data.

Spearman correlation across models and corpora shows that medical-specific datasets significantly outperform general-purpose Wikipedia embeddings.

As illustrated in Figure 1, Wikipedia-trained models consistently underperformed on clinical tasks. In contrast, embeddings trained on medical manuscripts (OA-All) and case reports (OA-CR) were highly competitive with those trained on restricted hospital notes. For the downstream task of mortality prediction, a 300-dimension Word2Vec model trained on OA-All achieved the best performance with a Brier Score of 0.13.

Discussion: The fastText Advantage and Its Quirk

A key technical insight was the superior performance of fastText in lexicographic coverage. By breaking words into sub-word "n-grams," fastText avoids the "out-of-vocabulary" problem, allowing it to interpret rare medical terms or typos. However, this sub-word reliance can lead to a specific quirk in how the model organizes information.

t-SNE visualization of embedding spaces. FastText (top) creates significantly tighter clusters based on sub-word patterns compared to GloVe (middle) and Word2Vec (bottom).

As shown in Figure 6, fastText tends to create extremely tight clusters. While beneficial for coverage, it can sometimes group words based on spelling similarity rather than deep semantic meaning (e.g., "heart" and "hearth"). This highlights a trade-off between linguistic coverage and semantic precision.

Conclusion

Word embeddings trained on open-access case reports provide a powerful, privacy-preserving, and computationally efficient tool for medical AI. By leveraging these "lightweight" models, researchers can conduct effective clinical NLP without the legal and ethical hurdles associated with protected health information.

Summary updated

Abstract

Abstract Rationale Word embeddings are used to create vector representations of text data but not all embeddings appropriately capture clinical information, are free of protected health information, and are computationally accessible to most researchers. Methods We trained word embeddings on published case reports because their language mimics that of clinical notes, the manuscripts are already de-identified by virtue of being published, and the corpus is much smaller than those trained on large, publicly available datasets. We tested the performance of these embeddings across five clinically relevant tasks and compared the results to embeddings trained on a large Wikipedia corpus, all publicly available manuscripts, notes from the MIMIC-III database using fastText, GloVe, and word2vec, and using different dimensions. Tasks included clinical applications of lexicographic coverage, semantic similarity, clustering purity, linguistic regularity, and mortality prediction. Results The embeddings trained using the published case reports performed as well as if not better on most tasks than those using other corpora. The embeddings trained using all published manuscripts had the most consistent performance across all tasks and required a corpus with 100 times as many tokens as the corpus comprised of only case reports. Embeddings trained on the MIMIC-III dataset had small but marginally better scores on the clustering tasks which was also based on clinical notes from the MIMIC-III dataset. Embeddings trained on the Wikipedia corpus, although containing almost twice as many tokens as all available published manuscripts, performed poorly compared to those trained on medical and clinical corpora. Conclusion Word embeddings trained on freely available published case reports performed well for most clinical task, are free of protected health information, and are small compared to commonly used embeddings trained on larger clinical and non-clinical corpora. The optimal corpus, dimension size, and which embedding model to use for a given task involves tradeoffs in privacy, reproducibility, performance, and computational resources.
BibTeX
@misc{flamholz2019word,
  author = {Zachary Flamholz and Lyle Ungar and Gary E. Weissman},
  title = {{Word embeddings trained on published case reports are lightweight, effective for clinical tasks, and free of protected health information}},
  year = {2019},
  doi = {10.1101/19013268},
  howpublished = {medRxiv},
}
Figure