From Text to Translation: Using Language Models to Prioritize Variants for Clinical Review

Weijiang Li, Xiaomin Li, Ethan Lavallee, Alice Saparov, Marinka Žitnik, Christopher A. CassamedRxiv 2024 · DOI

This research paper introduces ClinVar-BERT, a specialized language model designed to bridge the "translational gap" in genomic medicine by extracting clinical evidence from unstructured text reports.

TL;DR

Researchers fine-tuned a BioBERT-based model to "read" millions of unstructured clinical variant summaries from ClinVar, successfully identifying hidden evidence within "Variants of Uncertain Significance" (VUS). The model achieved an AUROC of 0.927 in classifying functional impact, providing a powerful tool for prioritizing which genetic variants should be fast-tracked for expert clinical review.

What Problem Does This Paper Address?

In the era of cheap genomic sequencing, the bottleneck has shifted from data generation to data interpretation. When a lab finds a rare genetic mutation, they often lack enough evidence to call it "pathogenic" (disease-causing) or "benign" (harmless). These variants are labeled Variants of Uncertain Significance (VUS).

ClinVar, a massive public archive, contains over 2.1 million plaintext summaries describing the evidence for these variants. However, this data is a "wild west" of unstructured text, lab-specific templates, and heterogeneous information. Manually reviewing these to find variants that are almost ready for reclassification is prohibitively slow. The authors sought to automate this "evidence-spotting" using Large Language Models (LLMs).

Methods

The authors developed a multi-stage pipeline to transform messy clinical text into a structured classification tool:

  1. Fuzzy Deduplication: Using MinHashLSH, they filtered out millions of nearly identical, template-based reports to ensure the model learned from diverse evidence rather than just memorizing lab-specific boilerplate.
  2. Sentence Classification (The "Evidence-Only" Filter): To prevent "data leakage," the team built a classifier to categorize sentences as Description (e.g., genomic coordinates), Conclusion (e.g., "this is pathogenic"), or Evidence (e.g., "found in 5 individuals with the disease"). They stripped away descriptions and conclusions, forcing the model to learn only from the underlying evidence.
  3. Fine-Tuning: They fine-tuned BioBERT (a model already pre-trained on PubMed) on this "evidence-only" dataset to classify variants into three categories: Pathogenic/Likely Pathogenic (P/LP), Benign/Likely Benign (B/LB), or VUS.
  4. Interpretability: They used Non-negative Matrix Factorization (NMF) via the Ecco library to visualize "neuron activation patterns," ensuring the model was actually looking at relevant clinical features like allele frequency and case studies.

Datasets

  • ClinVar: 2.4 million plaintext variant summaries used for training and initial testing.
  • MaveDB: Orthogonal functional screening data (Deep Mutational Scanning) used as a "ground truth" to validate if the model's text-based predictions matched actual biological behavior.
  • Computational Benchmarks: REVEL and AlphaMissense scores were used to further validate the model's performance against existing state-of-the-art algorithmic predictions.

Key Insights and Ideas

  • Stripping Conclusions is Vital: Including the lab's final verdict in the training text makes the task trivial for an LLM but useless for discovery. By removing the "answers," the researchers forced the model to develop a deeper "understanding" of what constitutes clinical evidence.
  • Emergent Evidence Clustering: Without explicit instruction, the model’s internal neurons began to cluster specific types of evidence. Visualization showed distinct "concepts" for population frequency, computational scores, and literature citations.
  • VUS Sub-classification: The model proved that many variants currently labeled as VUS in ClinVar actually contain enough textual evidence to suggest they are likely pathogenic or benign, effectively "pre-scoring" them for human experts.

Key Results

  • High Discriminative Power: ClinVar-BERT achieved an AUROC of 0.927 when distinguishing between variants that result in loss-of-function versus those with uncertain effects.
  • Biological Validation: The model’s predictions on VUS variants showed highly significant correlations with independent functional assay data for critical genes:
  • BRCA1 ($p = 1.90 \times 10^{-20}$)
    • TP53 ($p = 1.14 \times 10^{-47}$)
    • PTEN ($p = 3.82 \times 10^{-7}$)
  • Superiority over Base Models: Fine-tuning on ClinVar text led to massive performance gains; the base BioBERT model (without ClinVar tuning) achieved an AUROC of only ~0.51 on the same task.

Limitations and Future Work

  • Harmonization: Some variants have conflicting reports from different labs; the current model doesn't yet "reconcile" these into a single consensus.
  • Template Bias: Despite deduplication, the increasing use of standardized templates by labs might still lead to the model picking up on structural cues rather than pure evidence.
  • Circularity: There is a risk that some clinical summaries already incorporate the functional or computational data used for validation, though the authors attempted to mitigate this by using very recently published datasets.
  • Future Direction: The authors suggest moving toward extracting specific types of evidence (e.g., "this is a PM2 evidence code") to help expert panels like ClinGen automate their workflows.

Summary updated

Abstract

These findings suggest that ClinVar-BERT is capable of discerning evidence from diagnostic reports and can be useful for prioritizing variants for re-assessment by diagnostic laboratories and expert curation panels.
BibTeX
@misc{li2024from,
  author = {Weijiang Li and Xiaomin Li and Ethan Lavallee and Alice Saparov and Marinka Žitnik and Christopher A. Cassa},
  title = {{From Text to Translation: Using Language Models to Prioritize Variants for Clinical Review}},
  year = {2024},
  doi = {10.1101/2024.12.31.24319792},
  howpublished = {medRxiv},
}
Figure