exKidneyBERT: a language model for kidney transplant pathology reports and the crucial role of extended vocabularies

Tiancheng Yang, Ilia Sucholutsky, Kuang-Yu Jen, Matthias SchonlauPeerJ Comput. Sci 2024 · DOI

Introduction

Pathology reports represent the "gold standard" for clinical diagnosis, yet they are predominantly stored as unstructured text. Manually extracting data from these reports for clinical studies is a labor-intensive process prone to error. While Large Language Models (LLMs) like BERT have transformed natural language processing, they encounter significant obstacles in specialized medical subdomains such as renal transplant pathology.

Two primary hurdles exist: the Data Scarcity Trap, where standard BERT models require billions of words to learn a domain that may only have thousands of available reports, and the Sub-word Problem. BERT's tokenizer often breaks technical terms into fragments (e.g., "interstitial" becomes "inter," "##st," "##iti," and "##al"), causing these terms to lose their clinical significance and making it difficult for the model to grasp specific diagnostic criteria.

Methodology

To address these challenges, researchers developed exKidneyBERT. The study compared three model iterations to identify the primary drivers of performance:

  1. Clinical BERT: A baseline model pre-trained on general medical notes.
  2. Kidney BERT: Clinical BERT further pre-trained on a small corpus of 3,428 renal pathology reports (approximately 1.5 million words).
  3. exKidneyBERT: A novel approach where six crucial keywords—interstitial, fibrosis, tubular, atrophy, T-cell, and antibody—were manually added to the model’s vocabulary before repeating pre-training on the kidney-specific corpus.

By incorporating these words directly into the tokenizer's dictionary, the model treats them as single, meaningful concepts rather than fragmented sub-words.

Experimental Setup

The study utilized a dataset of 3,428 renal transplant pathology reports from the University of California, Davis, covering light microscopy, immunofluorescence, and electron microscopy. The models were evaluated on two primary clinical tasks: Information Retrieval (IR) to identify rejection types (TCMR and ABMR) and classification to determine the grade of interstitial fibrosis and tubular atrophy (IFTA).

Results and Discussion

The exKidneyBERT model demonstrated that in niche domains, vocabulary extension is more critical than raw data volume. While Kidney BERT showed minimal improvement over the baseline, exKidneyBERT achieved dramatic gains across all metrics.

Information Retrieval Performance

In IR tasks, exKidneyBERT significantly outperformed baselines by anchoring on specific technical terms. For Antibody-Mediated Rejection (ABMR), it achieved an 83.3% word overlap ratio, nearly doubling Clinical BERT's 46.1%. For T-cell Mediated Rejection (TCMR), it reached a 79.2% overlap ratio.

Figure 5: The architecture of Kidney BERT configured for Information Retrieval tasks, utilizing a linear layer to identify start and end spans within the text.

Classification and Grading

The model achieved a 95.8% exact match rate for IFTA grading. In predicting the general presence of rejection, exKidneyBERT reached an F1-score of 0.800, the highest among all tested models.

Extracted figure from page 8 of exKidneyBERT: a language model for kidney transplant pathology reports and the crucial role of extended vocabularies

These results challenge the conventional wisdom that more compute and larger datasets are the only paths to improvement. Instead, they highlight the power of targeted vocabulary extension in specialized fields.

Conclusion and Future Work

While exKidneyBERT proves highly effective, its current vocabulary extension is tailored to specific clinical questions. Future research may focus on automating the identification of significant "out-of-bag" technical terms to reduce manual selection. Additionally, while the model performs exceptionally well on a 1.5-million-word corpus, scaling to larger datasets as they become available remains a promising avenue for further performance enhancement.

Summary updated

BibTeX
@article{yang2024exkidneybert,
  author = {Tiancheng Yang and Ilia Sucholutsky and Kuang-Yu Jen and Matthias Schonlau},
  title = {{exKidneyBERT: a language model for kidney transplant pathology reports and the crucial role of extended vocabularies}},
  journal = {PeerJ Comput. Sci.},
  year = {2024},
  doi = {10.7717/peerj-cs.1888},
}
Figure