Enhancer Recognition: A Transformer Encoder-Based Method with WGAN-GP for Data Augmentation

Tianyu Feng, Tao Hu, Wenyu Liu, Yang ZhangInternational Journal of Molecular Sciences 2023 · DOI

TL;DR

This paper introduces a novel deep learning approach for identifying DNA enhancers and predicting their strength, crucial for understanding gene regulation and disease. By combining a Transformer encoder for sequence analysis with a Wasserstein GAN with Gradient Penalty (WGAN-GP) for data augmentation, the method effectively tackles the challenge of limited biological data, significantly improving prediction accuracy.

What Problem Does This Paper Address?

Identifying and understanding the function of enhancers – non-coding DNA sequences that regulate gene expression – is vital for deciphering gene regulatory networks and their role in health and disease. Traditional methods rely on manual feature engineering, which is laborious and not scalable. While deep learning offers promise, it typically requires large, high-quality datasets, which are often difficult and time-consuming to acquire in biological research. This scarcity can lead to overfitting and hinder model generalization. This paper aims to overcome these limitations by developing a robust enhancer recognition model that can perform well even with limited initial data.

Methods

The core of this work lies in two key innovations:

  • Transformer Encoder for Sequence Analysis: The authors employ a Transformer encoder as the primary architecture. Unlike traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs) that process sequences sequentially or locally, the Transformer's self-attention mechanism allows it to consider the entire input sequence simultaneously. This enables it to capture long-range dependencies and complex relationships within DNA sequences more effectively, leading to better feature extraction.
  • WGAN-GP for Data Augmentation: To address the issue of small datasets, the researchers utilize a Wasserstein GAN with a Gradient Penalty (WGAN-GP). This advanced generative model creates synthetic DNA sequences that mimic the statistical properties of real enhancer and non-enhancer data. This augmentation significantly expands the training dataset, mitigating overfitting and improving the model's ability to generalize to unseen data.

Additionally, the paper incorporates a k-mer coding layer to represent DNA sequences. This method divides sequences into overlapping "words" of length 'k' (e.g., 3-mers, 5-mers), preserving local sequence information more effectively than simple one-hot encoding.

Datasets

The study utilizes a benchmark dataset previously established by Liu et al. [18]. This dataset consists of:

  • Training Set: 1484 enhancers (742 strong, 742 weak) and 1484 nonenhancers.
  • Independent Validation Set: 200 enhancers (100 strong, 100 weak) and 200 nonenhancers.

The WGAN-GP is used to generate approximately 20,000 additional synthetic samples to augment the training data.

Key Insights and Ideas

  • Transformer's Superiority for Long-Range Dependencies: The Transformer's self-attention mechanism is particularly well-suited for DNA sequences, which can have regulatory elements separated by significant distances. This allows it to capture crucial interactions that simpler models might miss.
  • WGAN-GP as a Powerful Data Augmentation Tool: The WGAN-GP proves effective in generating realistic synthetic DNA data, significantly boosting model performance when real data is scarce. This highlights the potential of advanced GANs in biological sequence analysis.
  • K-mer Encoding Enhances Feature Representation: Using k-mers as an input representation for DNA sequences provides a more informative way to capture local sequence motifs compared to basic encoding methods, leading to improved accuracy.
  • Distinguishing Enhancers vs. Predicting Strength: The study suggests that classifying enhancers from nonenhancers is generally easier than predicting the precise strength of an enhancer, as reflected in the performance metrics for the two tasks.

Key Results

The proposed method demonstrates significant improvements across various performance metrics:

  • Task 1 (Enhancer/Nonenhancer Classification): On the independent validation set, the model achieved an average Accuracy (ACC) of 0.7746 and an Area Under the ROC Curve (AUC) of 0.841.
  • Task 2 (Enhancer Strength Prediction): On the independent validation set, the model achieved an average ACC of 0.8325 and an AUC of 0.917.
  • Comparison with Existing Methods: The proposed Transformer-based method with WGAN-GP augmentation outperformed several existing methods, including iEnhancer-2L, iEnhancer-EL, iEnhancer-CNN, and iEnhancer-GAN, particularly on the independent validation set. For instance, in enhancer/nonenhancer classification (task 1), the proposed method achieved a higher SN (0.815) on the validation set compared to others.

Limitations and Future Work

  • Data Diversity Limitation of WGAN-GP: While WGAN-GP effectively expands the dataset and improves performance, the authors acknowledge that it primarily amplifies existing data features rather than introducing entirely new diversity. This is noted as a limitation.
  • Future Work: The authors plan to focus on more advanced feature extraction techniques from the data. They also aim to apply their model to research on enhancer-associated disease mechanisms and assist in drug development.

Summary updated

Abstract

Enhancers are located upstream or downstream of key deoxyribonucleic acid (DNA) sequences in genes and can adjust the transcription activity of neighboring genes. Identifying enhancers and determining their functions are important for understanding gene regulatory networks and expression regulatory mechanisms. However, traditional enhancer recognition relies on manual feature engineering, which is time-consuming and labor-intensive, making it difficult to perform large-scale recognition analysis. In addition, if the original dataset is too small, there is a risk of overfitting. In recent years, emerging methods, such as deep learning, have provided new insights for enhancing identification. However, these methods also present certain challenges. Deep learning models typically require a large amount of high-quality data, and data acquisition demands considerable time and resources. To address these challenges, in this paper, we propose a data-augmentation method based on generative adversarial networks to solve the problem of small datasets. Moreover, we used regularization methods such as weight decay to improve the generalizability of the model and alleviate overfitting. The Transformer encoder was used as the main component to capture the complex relationships and dependencies in enhancer sequences. The encoding layer was designed based on the principle of k-mers to preserve more information from the original DNA sequence. Compared with existing methods, the proposed approach made significant progress in enhancing the accuracy and strength of enhancer identification and prediction, demonstrating the effectiveness of the proposed method. This paper provides valuable insights for enhancer analysis and is of great significance for understanding gene regulatory mechanisms and studying disease correlations.
BibTeX
@article{feng2023enhancer,
  author = {Tianyu Feng and Tao Hu and Wenyu Liu and Yang Zhang},
  title = {{Enhancer Recognition: A Transformer Encoder-Based Method with WGAN-GP for Data Augmentation}},
  journal = {International Journal of Molecular Sciences},
  year = {2023},
  doi = {10.3390/ijms242417548},
}
Figure