A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models

Hirofumi Tsuruta, Hiroyuki Yamazaki, Ryota Maeda, Ryotaro Tamura, Akihiro ImuraNeurIPS 2024

Predicting whether a specific antibody will successfully bind to its target antigen is often considered the "holy grail" of computational immunology. While large language models (LLMs) have made significant strides in protein sequence restoration and generation, true binding prediction requires high-quality, experimentally verified data.

Presented at NeurIPS 2024, this paper introduces a massive new resource for antibody language models (AbLMs) to bridge this gap: AVIDa-SARS-CoV-2, a large-scale labeled dataset of nanobody interactions, and VHHCorpus-2M, a pre-training corpus of over two million unique sequences.

The Problem: A Dearth of Ground-Truth Labels

In recent years, researchers have applied LLMs to biology by treating protein sequences like sentences, where amino acids act as "words." Databases like the Observed Antibody Space (OAS) provide billions of sequences for pre-training. However, there is a severe shortage of precise, sequence-level labeled data for binding.

Most existing benchmarks rely on noisy labels—for instance, assuming all antibodies found in a COVID-19 patient bind to the virus, even though many are unrelated. This lack of ground-truth data makes it difficult to train and evaluate models for actual drug discovery. The authors address this by providing a dataset where binding is experimentally verified at the sequence level.

A "Wet Lab to Dry Lab" Pipeline

To generate this high-quality data, the researchers utilized a comprehensive experimental pipeline:

  1. Immunization: Two alpacas were immunized with various SARS-CoV-2 spike proteins. Alpacas naturally produce VHHs (nanobodies)—single-chain antibodies that are smaller, more stable, and easier to engineer than standard human antibodies.
  2. Affinity Selection (Biopanning): The researchers used cell and bead "biopanning" to physically separate VHHs that bind to specific antigens from those that do not.
  3. Next-Generation Sequencing (NGS): By sequencing the VHH pools before and after selection, they statistically determined which sequences were enriched (binders) or depleted (non-binders).
Experimental pipeline from alpaca immunization to data labeling

The Datasets

The pipeline yielded two primary open-source resources:

  • AVIDa-SARS-CoV-2: A labeled dataset containing 77,003 samples (22,002 binders and 55,001 non-binders). It covers interactions with 13 different antigen types, including the original wild-type strain and major variants like Delta and Omicron.
  • VHHCorpus-2M: A pre-training collection of 2,040,988 unique VHH sequences derived from five alpacas. Unlike broader databases, this corpus consists entirely of full-length VHH sequences, representing the smallest functional units for binding.

Key Insights: Specificity and Diversity

The dataset reveals several critical insights into antibody behavior and model training:

  • The Extreme Specificity of VHHs: Even VHHs with greater than 90% sequence identity can exhibit completely different binding profiles across viral variants. Binding is highly sensitive to tiny mutations, making this a challenging but essential benchmark for AI.
  • Individual Differences Matter: The data shows that different alpacas produce distinct "clusters" of antibodies, even when immunized with the exact same antigen. This diversity highlights the need to train models capable of generalizing across different immune repertoires.
Clustering of VHH sequences showing distinct binding profiles and individual alpaca repertoires

Model Performance and the "65% Ceiling"

The authors pre-trained a RoBERTa-based model, VHHBERT, on their new 2-million-sequence corpus and benchmarked it against nine baseline models, including general protein models (ESM-2) and antibody-specific models (AntiBERTa2).

  • Domain-Specific Pre-training Wins: Models pre-trained specifically on antibodies consistently outperformed general-purpose protein models. For example, ESM-2 650M achieved an F1-score of 0.598, lagging behind specialized models despite having significantly more parameters.
  • VHHBERT Efficiency: Despite being trained on only 2 million sequences (compared to the 824 million used for AntiBERTa2), VHHBERT performed competitively with an F1-score of 0.608.
  • Top Performer: AntiBERTa2-CSSP, a multimodal model using both sequence and structure, achieved the best overall results (F1-score: 0.652, AUPRC: 0.690).

Limitations and Future Work: Currently, even the best models peak at around a 65% F1-score for binding prediction, indicating that mapping the complex physical interactions of binding remains a difficult open challenge. Furthermore, the labeled dataset is derived from only two alpacas, which may introduce individual-induced biases. Future work will focus on expanding dataset diversity and exploring new model architectures to push past the current performance ceiling.

Summary updated

BibTeX
@inproceedings{tsuruta2024sars,
  author = {Hirofumi Tsuruta and Hiroyuki Yamazaki and Ryota Maeda and Ryotaro Tamura and Akihiro Imura},
  title = {{A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure