KansformerEPI: a deep learning framework integrating KAN and transformer for predicting enhancer-promoter interactions

Tianjiao Zhang, Saihong Shao, Hongfei Zhang, Zhongqian Zhao, Xingjie Zhao, Zhao-Xiang Zhang, Zhenxing Wang, Guohua WangBriefings in Bioinformatics 2025 · DOI

TL;DR

This paper introduces KansformerEPI, a novel deep learning framework that integrates Kolmogorov-Arnold Networks (KANs) and Transformers to predict enhancer-promoter interactions (EPIs) across multiple cell types. By effectively capturing complex, non-linear relationships between genomic features, KansformerEPI achieves state-of-the-art performance and offers a more scalable and generalizable approach to understanding gene regulation.

What Problem Does This Paper Address?

Gene regulation is a fundamental biological process, and understanding how enhancers (regions that boost gene expression) interact with promoters (the "switches" that initiate transcription) is crucial for deciphering cellular function and disease mechanisms. These enhancer-promoter interactions (EPIs) can occur over long genomic distances, making their accurate prediction a significant challenge. Current computational methods often struggle with:

  • Limited Scope: Many models are trained on single cell lines, requiring retraining for each new tissue or cell type, which is inefficient and data-intensive.
  • Non-linear Relationships: Existing methods often fail to fully capture the complex, non-linear interactions between various genomic features (like DNA sequence and epigenetic marks), leading to suboptimal prediction accuracy.
  • Data Imbalance: Datasets for EPI prediction are typically highly imbalanced, with far more non-interacting pairs than interacting ones, posing a challenge for model training.

This paper aims to address these limitations by developing a more accurate, scalable, and generalizable model for predicting EPIs across diverse biological contexts.

Methods

The core innovation of KansformerEPI lies in its Kansformer encoder, which combines two powerful deep learning architectures:

  • Kolmogorov-Arnold Networks (KANs): Unlike traditional Multi-Layer Perceptrons (MLPs) that use fixed activation functions, KANs introduce learnable activation functions within their weights. This allows KANs to more effectively capture intricate non-linear relationships between features, offering greater flexibility and adaptability.
  • Transformer: This architecture, particularly its multi-head self-attention mechanism, excels at capturing long-range dependencies within sequential data.

KansformerEPI integrates these components to process both sequence and epigenetic features. The framework first uses a Convolutional Neural Network (CNN) for initial feature extraction, followed by a Bidirectional Long Short-Term Memory (BiLSTM) layer to capture temporal dependencies. The output of the BiLSTM is then fed into the Kansformer encoder, which uses alternating Multi-Head Attention (MHA) layers and KAN layers. This design allows the model to simultaneously capture local patterns, long-range dependencies, and complex non-linear feature interactions.

Additionally, the model incorporates a dual loss function: one for predicting the probability of EPI (binary cross-entropy) and another for predicting the distance between enhancers and promoters (mean squared error).

Datasets

The study utilized the BENGI dataset, which contains experimentally validated enhancer-gene interaction pairs.

  • Training Data: A combination of data from two cell lines, GM12878 and HeLa-S3, was used for training.
  • Independent Test Data: The model was evaluated on four additional cell lines: HMEC, IMR90, K562, and NHEK.
  • Methylation Data: Methylation features were obtained from a previous study [24] for GM12878, HeLa-S3, IMR90, and K562 cell lines.

The paper also discusses the use of an undersampling strategy to address dataset imbalance, creating a 1:1 ratio of positive to negative samples for training.

Key Insights and Ideas

  • KANs for Non-linearity: The integration of KANs is a key differentiator, enabling the model to move beyond the limitations of fixed activation functions in MLPs and better model the complex, non-linear nature of biological interactions.
  • Global EPI Prediction: By training on multiple cell lines and testing across others, KansformerEPI demonstrates the feasibility and effectiveness of a "global" EPI prediction model, reducing the need for tissue-specific model retraining.
  • Feature Importance Varies: Analysis revealed that different genomic features (like DNase-I signals, H3K4me1, and H3K4me3) play varying roles in EPI prediction depending on their genomic context (enhancer vs. promoter regions).
  • Multi-Input (MI) Dominance: The study found that the most prevalent regulatory relationship pattern is "Multi-Input" (MI), where a single promoter is regulated by multiple enhancers. This pattern also showed the highest prediction accuracy, suggesting its biological significance.

Key Results

  • State-of-the-Art Performance: KansformerEPI consistently outperformed existing methods like TransEPI, TargetFinder, SPEID, 3DPredictor, and DeepTACT across multiple cell lines in terms of AUROC and AUPR metrics. For example, it showed a ~7-8% improvement in AUC for IMR90 and NHEK compared to TransEPI.
  • Improved Accuracy and Stability: The model demonstrated enhanced accuracy and stability in EPI predictions across diverse datasets.
  • Methylation Impact: The impact of methylation features was mixed; it showed a slight enhancement in AUPR for IMR90 but a slight decrease in both AUROC and AUPR for K562. This suggests methylation's role might be context-dependent.
  • Balanced Datasets Improve Performance: Training on balanced datasets (1:1 ratio of positive to negative samples) consistently led to better performance compared to training on imbalanced data.
  • DKK1 Gene Validation: The model successfully predicted experimentally validated EPIs associated with the DKK1 gene, achieving significantly higher interaction probability scores than TransEPI.

Limitations and Future Work

  • Methylation Impact Variability: The inconsistent impact of methylation features across cell lines warrants further investigation. The authors suggest that current data might be insufficient to definitively prove its universal benefit.
  • Transformer Memory Intensity: The self-attention mechanism in Transformers is memory-intensive, requiring storage of relationships between all sequence elements.
  • KAN Training Speed: KANs, while powerful, can be significantly slower to train than MLPs.

Future work will focus on:

  • Improving the training efficiency of KANs.
  • Developing innovative solutions to reduce the memory consumption of Transformers.
  • Further exploring the role and integration of methylation features.

Summary updated

BibTeX
@article{zhang2025kansformerepi,
  author = {Tianjiao Zhang and Saihong Shao and Hongfei Zhang and Zhongqian Zhao and Xingjie Zhao and Zhao-Xiang Zhang and Zhenxing Wang and Guohua Wang},
  title = {{KansformerEPI: a deep learning framework integrating KAN and transformer for predicting enhancer-promoter interactions}},
  journal = {Briefings in Bioinformatics},
  year = {2025},
  doi = {10.1093/bib/bbaf272},
}
Figure