TFBS-Finder: Deep Learning-based Model with DNABERT and Convolutional Networks to Predict Transcription Factor Binding Sites

Nimisha Ghosh, Pratik Dutta, Daniele Santoni

Abstract

Transcription factors are proteins that regulate the expression of genes by binding to specific genomic regions known as Transcription Factor Binding Sites (TFBSs), typically located in the promoter regions of those genes. Accurate prediction of these binding sites is essential for understanding the complex gene regulatory networks underlying various cellular functions. In this work, we have developed a deep learning model which uses pre-trained DNABERT, a Convolutional Neural Network (CNN) module, a Modified Convolutional Block Attention Module (MCBAM), a Multi-Scale Convolutions with Attention (MSCA) module and an output module. The pre-trained DNABERT is used for sequence embedding, thereby capturing the long-term dependencies in the DNA sequences while the CNN, MCBAM and MSCA modules are useful in extracting higher-order local features. TFBS-Finder is trained and tested on 165 ENCODE ChIP-seq datasets. We have also performed ablation studies as well as cross-cell line validations and comparisons with other models. The experimental results show the superiority of the proposed method in predicting TFBSs compared to the existing methodologies.

Introduction

Transcription factors (TFs) act as critical regulators of the genetic code, binding to specific DNA sequences called Transcription Factor Binding Sites (TFBSs) to control gene expression. Mapping these sites is vital for understanding biological processes and disease mechanisms. While experimental methods like ChIP-seq are highly accurate, they are costly and time-consuming. Computational models offer a scalable alternative, yet they must overcome the challenge of capturing both short, local motifs and long-range genomic dependencies. Existing models often struggle to balance these two scales effectively.

Methodology

The TFBS-Finder architecture is a five-module pipeline designed to integrate global context-awareness with specialized local feature extraction.

The overall pipeline of the TFBS-Finder model.

The model's components include:

  1. DNABERT: A pre-trained transformer that treats DNA as a language, providing embeddings that capture global context and long-range dependencies.
  2. CNN Module: A convolutional layer that extracts high-order local features from the DNABERT embeddings.
  3. Modified Convolutional Block Attention Module (MCBAM): This module applies spatial and channel attention to highlight important sequence regions and features. The authors found that applying spatial attention before channel attention optimized performance.
  4. Multi-Scale Convolutions with Attention (MSCA): Uses varying kernel sizes to detect patterns of different lengths simultaneously.
  5. Output Module: Fuses features from the MCBAM and MSCA modules to generate the final prediction.
Detailed architecture of the spatial (a) and channel (b) attention modules.

Experimental Setup and Datasets

The model was rigorously evaluated using 165 ENCODE ChIP-seq datasets, covering 29 transcription factors across 32 human cell lines. Positive samples were 101-bp DNA sequences, while negative samples were generated by shuffling positive sequences while preserving dinucleotide frequencies to maintain biological realism.

Results and Discussion

TFBS-Finder consistently outperformed existing state-of-the-art models. It achieved an average accuracy of 0.930, a PR-AUC of 0.961, and a ROC-AUC of 0.961. Notably, it surpassed the previous leading model, BERT-TFBS, by 7.9% in accuracy.

Ablation studies confirmed that the combination of Transformers and CNN-based attention modules is highly effective; DNABERT handles global logic, while the attention modules act as refined local motif detectors. Furthermore, the model demonstrated robustness in cross-cell line validations, maintaining high performance (ROC-AUC > 0.93) across different cell types, suggesting it learns general biological rules rather than dataset-specific artifacts.

Conclusion

TFBS-Finder represents a significant advancement in the computational prediction of transcription factor binding sites. By leveraging the strengths of both transformer-based global embeddings and multi-scale attention-driven local feature extraction, it achieves superior accuracy and robustness. Future research will investigate the incorporation of 3D DNA shape information to further improve predictive performance.

Summary updated

Abstract

Transcription factors are proteins that regulate the expression of genes by binding to specific genomic regions known as Transcription Factor Binding Sites (TFBSs), typically located in the promoter regions of those genes. Accurate prediction of these binding sites is essential for understanding the complex gene regulatory networks underlying various cellular functions. In this regard, many deep learning models have been developed for such prediction, but there is still scope of improvement. In this work, we have developed a deep learning model which uses pre-trained DNABERT, a Convolutional Neural Network (CNN) module, a Modified Convolutional Block Attention Module (MCBAM), a Multi-Scale Convolutions with Attention (MSCA) module and an output module. The pre-trained DNABERT is used for sequence embedding, thereby capturing the long-term dependencies in the DNA sequences while the CNN, MCBAM and MSCA modules are useful in extracting higher-order local features. TFBS-Finder is trained and tested on 165 ENCODE ChIP-seq datasets. We have also performed ablation studies as well as cross-cell line validations and comparisons with other models. The experimental results show the superiority of the proposed method in predicting TFBSs compared to the existing methodologies. The codes and the relevant datasets are publicly available at https://github.com/NimishaGhosh/TFBS-Finder/.
BibTeX
@misc{ghosh2025tfbs,
  author = {Nimisha Ghosh and Pratik Dutta and Daniele Santoni},
  title = {{TFBS-Finder: Deep Learning-based Model with DNABERT and Convolutional Networks to Predict Transcription Factor Binding Sites}},
  year = {2025},
  eprint = {2502.01311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01311},
}
Figure