iEnhancer-CLA: Self-attention-based interpretable model for enhancers and their strength prediction

Lijun Cai, Xiangrong Ren, Xiangzheng Fu, Mingyu Gao, Peng Wang, Junling Xu, Wei Liu, Zejun Li, Xiangxiang ZengbioRxiv 2021 · DOI

TL;DR

This paper introduces iEnhancer-CLA, a novel deep learning model that accurately predicts enhancers and their strengths while offering unprecedented interpretability. By combining multi-scale CNNs with a self-attention mechanism, the model not only achieves state-of-the-art performance but also reveals key sequence features that distinguish enhancers from non-enhancers, paving the way for deeper biological insights.

What Problem Does This Paper Address?

Enhancers are crucial non-coding DNA elements that regulate gene transcription, playing a vital role in eukaryotic development. While computational methods have made strides in predicting enhancers, they often suffer from a "black box" problem: they achieve good performance but lack interpretability. This means it's difficult to understand why a sequence is predicted as an enhancer or what specific sequence features contribute to its strength. This lack of interpretability hinders our ability to gain deeper biological insights into enhancer function and can limit the ability to improve models further. Furthermore, enhancers are a diverse group with varying strengths, and accurately classifying these subtypes remains a challenge.

Methods

The iEnhancer-CLA model tackles these challenges with a multi-layered deep learning architecture:

  • Multi-scale Convolutional Neural Networks (CNNs): To automatically learn sequence features, the model employs CNNs with filters of different lengths (15 and 25 nucleotides). This allows it to capture patterns at various scales within the DNA sequence.
  • Bidirectional Long Short-Term Memory (BLSTM) Layers: Following the CNNs, BLSTM layers are used to process the learned features in both forward and backward directions, capturing sequential dependencies.
  • Multi-head Self-Attention Mechanism: This is a core innovation. The self-attention mechanism allows the model to weigh the importance of different parts of the sequence when making a prediction. By using multiple "heads," it can attend to different aspects of the sequence simultaneously, effectively capturing "multibody effects" or interactions between distant sequence elements. This mechanism is key to the model's interpretability.
  • Bayesian Optimization: To avoid manual hyperparameter tuning, which can be time-consuming and prone to bias, the researchers employed Bayesian optimization to find the optimal global hyperparameters for the model.
  • Decoupling for Interpretability: A key aspect of iEnhancer-CLA is its ability to decouple its components. By analyzing the weights generated by the CNNs and the self-attention layers, the researchers can visualize and understand which sequence motifs and base combinations are most influential in predicting enhancers and their strengths.

Datasets

The study utilized a benchmark dataset that was divided into training and independent test sets, following the methodology of a previous study (Liu et al., 2016). The paper does not mention introducing a new benchmark dataset.

Key Insights and Ideas

  • Interpretability through Self-Attention: The self-attention mechanism isn't just for performance; it's a powerful tool for understanding why the model makes its predictions. By visualizing attention weights, researchers can pinpoint critical regions and base patterns.
  • Sequence Feature Differences: The analysis revealed distinct base composition patterns: non-enhancers tend to have a more uniform distribution with enriched A and T bases, while enhancers, particularly strong ones, show uneven distributions with a higher enrichment of G and C bases. This finding offers valuable clues for future enhancer prediction models and understanding functional mechanisms.
  • CNNs Learn Motifs: Decoupling the CNN layers confirmed their effectiveness in learning sequence motifs, which, when combined with self-attention weights, provides a comprehensive view of sequence importance.
  • Bayesian Optimization for Robustness: Using Bayesian optimization for hyperparameter tuning ensures a more systematic and potentially globally optimal configuration, reducing reliance on manual adjustments.

Key Results

  • Performance Improvement: iEnhancer-CLA outperformed existing predictors in terms of accuracy for identifying enhancers and their strengths.
  • Enhanced Interpretability: The model successfully visualized enhancer motifs and key base signals by decoupling CNN modules and generating self-attention weights.
  • Biological Insights: The analysis revealed that non-enhancer sequences have a more uniform base distribution with enriched A and T, while enhancers, especially strong ones, exhibit uneven distributions with enriched G and C bases. Weak enhancers also showed enrichment of A and T.
  • Table 1 Comparison: In a comparison with existing two-layer models, iEnhancer-CLA achieved higher accuracy metrics (ACC, AUC, SP, MCC) for both the first layer (enhancer prediction) and the second layer (strength prediction), particularly for strong and weak enhancers. For instance, the strong enhancer prediction achieved an ACC of 78.50% and AUC of 84.69%.

Limitations and Future Work

  • Subtype Expansion: The authors acknowledge that enhancers are a diverse group with many subtypes beyond strong and weak. Future work could focus on developing models to distinguish and predict these additional subtypes.
  • Model Extension: The proposed multi-scale deep learning model could be extended to predict more enhancer subtypes, suggesting a pathway for future prediction development.
  • Further Sequence Analysis: The identified differences in base composition and distribution between enhancers and non-enhancers can serve as a guide for constructing subsequent models and for further revealing potential functional mechanisms of enhancers.

Summary updated

Abstract

Abstract Enhancer is a class of non-coding DNA cis-acting elements that plays a crucial role in the development of eukaryotes for their transcription. Computational methods for predicting enhancers have been developed and achieve satisfactory performance. However, existing computational methods suffer from experience-based feature engineering and lack of interpretability, which not only limit the representation ability of the models to some extent, but also make it difficult to provide interpretable analysis of the model prediction findings.In this paper, we propose a novel deep-learning-based model, iEnhancer-CLA, for identifying enhancers and their strengths. Specifically, iEnhancer-CLA automatically learns sequence 1D features through multiscale convolutional neural networks (CNN), and employs a self-attention mechanism to represent global features formed by multiple elements (multibody effects). In particular, the model can provide an interpretable analysis of the enhancer motifs and key base signals by decoupling CNN modules and generating self-attention weights. To avoid the bias of setting hyperparameters manually, we construct Bayesian optimization methods to obtain model global optimization hyperparameters. The results demonstrate that our method outperforms existing predictors in terms of accuracy for identifying enhancers and their strengths. Importantly, our analyses found that the distribution of bases in enhancers is uneven and the base G contents are more enriched, while the distribution of bases in non-enhancers is relatively even. This result contributes to the improvement of prediction performance and thus facilitates revealing an in-depth understanding of the potential functional mechanisms of enhancers. Author summary The enhancers contain many subspecies and the accuracy of existing models is difficult to improve due to the small data set. Motivated by the need for accurate and efficient methods to predict enhancer types, we developed a self-attention deep learning model iEnhancer-CLA, the aim is to be able to distinguish effectively and quickly between subspecies of enhancers and whether they are enhancers or not. The model is able to learn sequence features effectively through the combination of multi-scale CNN blocks, BLSTM layers, and self-attention mechanisms, thus improving the accuracy of the model. Encouragingly, by decoupling the CNN layer it was found that the layer was effective in learning the motif of the sequences, which in combination with the self-attention weights could provide interpretability to the model. We further performed sequence analysis in conjunction with the model-generated weights and discovered differences in enhancer and non-enhancer sequence characteristics. This phenomenon can be a guide for the construction of subsequent models for identifying enhancer sequences.
BibTeX
@misc{cai2021ienhancer,
  author = {Lijun Cai and Xiangrong Ren and Xiangzheng Fu and Mingyu Gao and Peng Wang and Junling Xu and Wei Liu and Zejun Li and Xiangxiang Zeng},
  title = {{iEnhancer-CLA: Self-attention-based interpretable model for enhancers and their strength prediction}},
  year = {2021},
  doi = {10.1101/2021.11.23.469658},
  howpublished = {bioRxiv},
}
Figure