The Need for Transfer Learning in CRISPR-Cas Off-Target Scoring

Pavan K. Kota, Yidan Pan, Hoang-Anh Vu, Mingming Cao, Richard G. Baraniuk, Gang BaobioRxiv 2021 · DOI

TL;DR

This paper introduces "TrueOT," a new benchmark dataset for evaluating CRISPR-Cas9 off-target scoring models, as existing datasets don't accurately reflect in vivo editing. They demonstrate that models trained on high-throughput "proxy" assays perform poorly on TrueOT, highlighting the need for transfer learning to bridge the gap between assay data and real-world biological outcomes.

What Problem Does This Paper Address?

CRISPR-Cas9 gene editing is a powerful tool, but unintended edits at "off-target" (OT) sites are a major concern, potentially leading to harmful mutations. To design safer CRISPR systems, computational models are used to predict which DNA sites a guide RNA (gRNA) might edit. However, a significant challenge is the lack of a standardized, high-quality dataset that accurately reflects in vivo (in living cells) editing. Many existing models are trained on data from high-throughput in vitro (in a lab dish) or proxy assays. The authors hypothesize that these proxy assays, while scalable, don't fully capture the complex biological nuances of in vivo editing, leading to models that don't generalize well to real-world scenarios. This makes it difficult to compare different scoring models and hinders the development of truly safe gene editing tools.

Methods

The core of the paper involves developing and evaluating Siamese convolutional neural networks (CNNs). Siamese networks are designed to compare pairs of inputs (in this case, gRNA-target DNA sequences) by passing them through identical network branches and then measuring the similarity or distance between their outputs.

Key methodological aspects include:

  • Data Encoding: Instead of relying on sequence alignment (which can be inconsistent), they use a simplified, alignment-independent approach by one-hot encoding nucleotides and left-padding sequences to a fixed length. This method aims to be robust to small insertions or deletions (indels) that can occur due to bulges.
  • Siamese Network Architectures: They explore several Siamese CNN architectures:
  • S1C: A simple model with a single convolutional and pooling layer.
  • S2C and S2C_Dense: More complex models with additional layers to investigate the impact of increased depth.
  • S1C_ut: An untrained version of S1C with random weights to understand the role of learned features versus basic pattern matching.
  • Hybrid S1C: A transfer learning approach where a pre-trained S1C model's convolutional layers are frozen, and a new dense layer is trained on a smaller dataset.
  • Transfer Learning: They specifically investigate transfer learning by training models on a "Proxy Dataset" and then fine-tuning or evaluating them on the "TrueOT" dataset. This allows them to test how well knowledge gained from one data source transfers to another.

Datasets

  • TrueOT Dataset (New Benchmark): This is the paper's primary contribution. It's a curated dataset of 1903 in vivo validated datapoints. The authors emphasize that these datapoints reflect actual mutation rates in living cells, making them a more accurate representation of editing outcomes than proxy assays.
  • Proxy Dataset: This dataset, comprising 3481 datapoints, was compiled from existing literature and predominantly includes data from high-throughput proxy assays like GUIDE-SEQ and HTGTS. This dataset was used for training the Siamese models.

Key Insights and Ideas

  • The "Proxy Gap": Models trained on proxy assays don't perform well on in vivo data. This is a critical finding, suggesting that the biological mechanisms captured by proxy assays are fundamentally different from those governing in vivo editing.
  • Simplicity Can Be Powerful: Surprisingly, the simplest Siamese model (S1C) with just one convolutional layer achieved state-of-the-art performance on the TrueOT dataset. This suggests that complex architectures aren't always necessary and can even be detrimental if trained on the wrong data.
  • Over-optimization on Proxy Data is Harmful: Increasing model complexity to improve performance on the Proxy Dataset often led to worse performance on TrueOT. This strongly indicates that optimizing for proxy assay data can lead models astray from predicting true in vivo behavior.
  • Transfer Learning is Crucial: Applying transfer learning techniques, particularly with the Hybrid S1C model, showed improved generalization to TrueOT, suggesting it's a promising avenue for developing models that bridge the gap between different data sources.
  • Need for a Standard Benchmark: The lack of a universally accepted benchmark like TrueOT makes it hard to compare models and hinders progress in the field.

Key Results

  • The S1C model achieved state-of-the-art performance on the TrueOT dataset, outperforming several existing rule-based and ML models.
  • More complex Siamese models (S2C, S2C_dense) showed improved performance on the Proxy Dataset but compromised performance on TrueOT.
  • The untrained S1C_ut model performed better on TrueOT than many trained baseline models, suggesting that basic mismatch counting with some frameshift tolerance (achieved via max pooling) is a strong baseline for in vivo prediction.
  • The S1C_Hybrid_TL model, utilizing transfer learning, demonstrated improved ensemble performance over the original S1C on TrueOT test splits.
  • CRISPR-NET, a bulge-aware model, showed an edge on bulge-excluded datapoints, while the S1C performed better on bulge-containing datapoints.

Limitations and Future Work

  • Dataset Size: The TrueOT dataset, while validated, is relatively small (1903 datapoints). This limits the ability to train very deep neural networks directly on it.
  • Epigenetic Factors: The study acknowledges that it does not consider epigenetic factors, which can influence gene editing outcomes and vary between cell types.
  • Future Work:
    • Expand TrueOT: The authors advocate for community-driven expansion of the TrueOT dataset with more in vivo-validated data.
  • Transfer Learning: Further research into transfer learning is recommended, especially as more data becomes available for TrueOT, to fine-tune deeper models.
  • Nuanced Subsets: As more in vivo data accumulates, TrueOT could be split into more specific subsets for particular applications or cell types.
  • Broader Community Adoption: The paper strongly urges the CRISPR community to adopt a standardized benchmark dataset like TrueOT for consistent model evaluation.

Summary updated

Abstract

Abstract Motivation The scalable design of safe guide RNA sequences for CRISPR gene editing depends on the computational “scoring” of DNA locations that may be edited. As there is no widely accepted benchmark dataset to compare scoring models, we present a curated “TrueOT” dataset that contains thoroughly validated datapoints to best reflect the properties of in vivo editing. Many existing models are trained on data from high throughput assays. We hypothesize that such models may suboptimally transfer to the low throughput data in TrueOT due to fundamental biological differences between proxy assays and in vivo behavior. We developed new Siamese convolutional neural networks, trained them on a proxy dataset, and compared their performance against existing models on TrueOT. Results Our simplest model with a single convolutional and pooling layer surprisingly exhibits state-of-the-art performance on TrueOT. Adding subsequent layers improved performance on a proxy dataset while compromising performance on TrueOT. We demonstrate improved generalization on TrueOT with a Siamese model of higher complexity when we apply transfer learning techniques. These results suggest an urgent need for the CRISPR community to agree upon a benchmark dataset such as TrueOT and highlight that various sources of CRISPR data cannot be assumed to be equivalent. Availability and Implementation Our code base and datasets are available on GitHub at github.com/baolab-rice/CRISPR_OT_scoring .
BibTeX
@misc{kota2021need,
  author = {Pavan K. Kota and Yidan Pan and Hoang-Anh Vu and Mingming Cao and Richard G. Baraniuk and Gang Bao},
  title = {{The Need for Transfer Learning in CRISPR-Cas Off-Target Scoring}},
  year = {2021},
  doi = {10.1101/2021.08.28.457846},
  howpublished = {bioRxiv},
}
Figure