Models that learn gene regulation from expression data increasingly compress thousands of genes into a few dozen latent "modules", groups of genes that rise and fall together, and then let a recurrent model predict how those modules push one another forward in time. A recurrent model here means one that carries a running internal state from one time step to the next. The compression is what lets these models scale. It also creates a failure mode. Several modules can end up encoding the same dominant background program, such as cell-cycle or stress variation, while the real regulatory signal is squeezed into too few units.
The usual fix is to make the modules look different, for example by penalizing similar gene loadings. In biology this is the wrong criterion. Genes that share pathways legitimately load on overlapping modules, and co-expression is often the mechanism rather than noise. We propose a different test of redundancy: two modules are redundant if knocking either one out in the model produces the same predicted downstream response. We turn that test into a training penalty and into a validation protocol. We then ask whether it makes the model's in-silico knockouts better predictors of real, held-out genetic perturbations.
Core Claim
In a modular recurrent model of gene expression, the right place to enforce diversity is the space of simulated intervention effects, not the space of weights. Concretely, let $r_k$ be the change in the model's predicted expression trajectory over a fixed horizon when module $k$ is clamped to zero, averaged over a minibatch of starting cell states and normalized to unit length. We penalize collinearity among these effect vectors with a determinant term:
$$ \mathcal{L} = \mathcal{L}_{\text{pred}} - \lambda \log\det\big(G + \epsilon I\big), \qquad G_{kl} = r_k^\top r_l . $$
Here $\mathcal{L}_{\text{pred}}$ is the usual trajectory or perturbation-response loss, $G$ is the $m \times m$ similarity matrix of module knockout effects, $m$ is the number of modules, $\lambda$ sets the penalty strength, and $\epsilon$ is a small ridge term. The determinant is large only when the knockout effects span many independent directions. This is the repulsive prior of Priors for Diversity in Generative Latent Variable Models, which placed a determinantal point process over latent vectors so that topics and clusters stop duplicating each other. The difference is what the prior acts on. We apply it to causal effect vectors rather than to latent parameters. Because $m$ is tens to about a hundred rather than thousands of genes, the cubic cost that paper flags as its scaling limit does not bite.
The prediction is specific. Compared with the same model trained with no diversity term or with weight-space diversity, interventional diversity should improve prediction of held-out perturbations. It should also leave within-module co-expression intact and produce modules whose knockout effects align with the measured response to perturbing their top-loading transcription factor.
Where This Sits
The closest work is Interpretable Neural ODEs for Gene Regulatory Network Discovery under Perturbations (PerturbODE). It supplies our backbone. PerturbODE is a continuous-time recurrent model: a neural ordinary differential equation, meaning the network defines how expression changes at each instant. Genes are projected into modules by a matrix $B$, passed through a sigmoid, and mapped back by a matrix $A$, with intervention and decay terms added. It scaled to the 812-gene TF Atlas benchmark where DCDI and Bicycle ran out of memory. The file notes two limitations we target: nothing prevents its modules from collapsing onto redundant background structure, and it is unstable with high variance on time series.
We keep the architecture and change what "distinct modules" means. This distinction matters. Modules can have orthogonal loadings in $B$ yet nearly collinear columns in $A$, so their downstream effects coincide and a weight penalty cannot see it. Conversely, a gene that genuinely belongs to two pathways forces overlapping loadings, which a weight penalty would wrongly punish.
The in-silico knockout idea comes from Deciphering gene regulation from gene expression dynamics using deep neural network. That work trained a recurrent simulator on the Drosophila gap-gene system and blocked individual links to read off activation and inhibition. It predicted 152 of 166 expression features across 13 unseen mutants. It also has three limits the file identifies:
- it only ran on 2 to 7 genes;
- redundant links were removed by post-hoc greedy pruning;
- its instantaneous step-to-step transitions produced artificial self-activation on every gene.
We move knockouts from a post-hoc analysis into the training objective and from single links to modules, which is what makes them tractable at transcriptome scale. We also track spurious self-loops as a diagnostic, because redundancy control may reduce the model's need for them.
Two lines of work constrain the design. Variance is a poor guide to usefulness: Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows that high-variance principal components often carry background nuisance. This argues against simply projecting genes onto top principal components first. Second, geneRNIB: a living benchmark for gene regulatory network inference found that Pearson correlation and GRNBoost2 beat multiomic pipelines and foundation models on perturbation prediction. That result sets a demanding bar, and any claim for a recurrent model must clear it on the same metrics.
Other work in the file shows that redundancy-removal ideas can help in biology, though none of it targets dynamic trans-regulation. Seq2Exp (Learning to Discover Regulatory Elements for Gene Expression Prediction) uses an information bottleneck to find sparse causal DNA regions. PMF-GRN (A Variational Inference Approach to Single-Cell Gene Regulatory Network Inference using Probabilistic Matrix Factorization) uses priors to break collinearity in a static factorization. PCGII (Information-incorporated gene network construction with FDR control) uses partial correlations with false-discovery-rate control. All three work on static data or cis-regulation. None of them uses simulated interventions as the criterion for redundancy.
Research Program
Training. We reimplement the PerturbODE-style modular dynamics and add the interventional-diversity term. Computing $r_k$ exactly costs $m$ extra forward rollouts per step. We use two cheaper routes:
- a short horizon with a random subset of modules per minibatch;
- a first-order approximation obtained by differentiating the rollout with respect to a module gate.
We check the approximation against exact rollouts on small cases.
Discrete-time variant. We also train a gated recurrent unit (GRU) with the same modular bottleneck, the discrete-time recurrent family the file covers. This tests whether the effect depends on continuous time.
Validation protocol. Each module is scored by one question: does its in-silico knockout, or clamp-up for overexpression data, predict the measured response to perturbing the transcription factor that loads most strongly on it? Those perturbations are held out of training.
This complements, rather than replaces, database-based checks against ChIP-Atlas and Gene Ontology. The file documents that such databases are biased and partly circular: geneRNIB found that pruning edges by prior binding databases removed valid context-specific interactions. Held-out interventions are the least circular evidence available.
Analysis. We measure four things:
- how many modules collapse onto the first few principal components of expression, as a redundancy count;
- whether within-module co-expression is preserved;
- the stability across random seeds that PerturbODE lacked;
- the frequency of learned self-activation.
Baselines
Do-nothing references. The first is predicting no change from control expression, which anchors every perturbation metric. The second is Pearson correlation as scored in geneRNIB, which that benchmark found hard to beat. Both are run through the geneRNIB evaluation code on the same splits.
Static competitor. GRNBoost2 is the strongest static method in geneRNIB. We run it with its default settings and the same held-out perturbations. geneRNIB reports that it outperformed complex methods but gives no numbers; no reported result found on Lacuna for GRNBoost2 on these datasets. We therefore rerun it ourselves.
Strongest dynamic method. This is unmodified PerturbODE, which reports the highest AUPRC on ChIP-Atlas binding recovery and higher recall than NO-TEARS and DCDFG on TF Atlas. No numeric value for that AUPRC appears in the file; no reported result found on Lacuna for PerturbODE's exact AUPRC. It is rerun with the same number of modules, optimizer, and training epochs as our model.
Ablations isolating the key ingredient. These are identical models whose only change is the diversity term:
- a determinantal penalty on the rows of $B$;
- the same penalty on the concatenated columns of $A$ and rows of $B$;
- a soft orthogonality penalty on $B$;
- the no-penalty model with the number of modules tuned on validation data.
Tuning budget. Every variant gets the same hyperparameter search budget over its penalty strength (or module count) and learning rate, selected on a validation split of perturbations disjoint from test.
What counts as winning. The interventional penalty wins if it beats unmodified PerturbODE and every weight-space ablation on held-out perturbation prediction, meaning geneRNIB's unseen-gene and target-prediction regression scores and its Wasserstein distance. The margin must exceed seed-to-seed spread over at least five seeds, on at least three of the five geneRNIB datasets plus TF Atlas. It must also at least match GRNBoost2 and Pearson.
Why not just…?
Why not just put a determinantal or orthogonality penalty on the module weights? This is the real threat. It is cheaper, and for many datasets it may do nearly as well. The mechanistic argument against it is the one above: a weight penalty sees neither redundancy created by collinear columns of $A$ nor legitimate overlap from genes shared between pathways.
That argument is testable, and the weight-space ablations are built to test it. On SERGIO simulations we plant both cases by construction:
- duplicated downstream programs with non-overlapping loadings;
- genes shared between two true modules.
If weight-space penalties match interventional diversity on real held-out perturbations as well, we will report that the cheaper method suffices. We will then recast the contribution as the held-out-intervention validation protocol.
Why not just use fewer modules? Shrinking the module count reduces duplication, but it also forces distinct regulatory programs to merge. The tuned-count ablation settles this directly. If the best-tuned count without any penalty equals our method, the penalty adds nothing.
Why not just reduce genes with principal components first? The eigencomponent-selection result shows that top-variance directions often carry background nuisance rather than task signal. Projection before training would also discard exactly the low-variance regulators that perturbation data reveal. We include a principal-component front end as a secondary baseline to confirm this on expression data.
Why not just use Pearson correlation or GRNBoost2, as geneRNIB suggests? They may win on static endpoint metrics, and we will report it if they do. Their outputs cannot simulate a knockout trajectory or assign a sign and direction. The file notes, citing Kolmogorov-Arnold Network for Gene Regulatory Network Inference, that tree ensembles lack sign and direction. That paper reports its continuous model beat such baselines by 5.40% to 28.37% AUROC on BEELINE. The fair comparison is on geneRNIB's own metrics, where these methods are strongest.
Proposed Experiments
Synthetic check with known answers. We use SERGIO's 400-gene simulated network with two planted redundancy structures (duplicated programs and shared genes). The metrics are edge-recovery AUPRC and a module-recovery score against the true modules. The decisive pattern is that interventional diversity recovers both structures, while weight penalties fail at least one.
Main real-data tests. We use TF Atlas (812 genes, transcription-factor overexpression) and the geneRNIB datasets Norman, Replogle, Adamson, Nakatake, and OPSCA. Transcription factors are held out entirely. We score the geneRNIB regression scores and Wasserstein distance, and ChIP-Atlas AUPRC for binding support. The decisive check is the per-module test of whether knockout effects predict held-out transcription-factor responses better than under the ablations.
Time-series stability. On the yeast heat-shock time series used by MERLIN and BiGGEsTS, we measure:
- seed variance of trajectory error, addressing PerturbODE's reported instability;
- Gene Ontology enrichment of modules, the standard functional check used by MERLIN;
- the fraction of genes relying on self-activation.
Small-scale sanity check. On the Drosophila gap-gene data, the model must reproduce the earlier recurrent simulator's level of mutant prediction, 152 of 166 features and boundary error of 4% of embryo length. This confirms that the added penalty does not harm a system where the true wiring is known.
Kill criterion. A null result on the main test counts as informative. If interventional diversity fails to beat both unmodified PerturbODE and the weight-space penalty on held-out perturbations, we will conclude that module redundancy is not what limits current recurrent gene-network models.
References
- Priors for Diversity in Generative Latent Variable Models
- Interpretable Neural ODEs for Gene Regulatory Network Discovery under Perturbations
- Deciphering gene regulation from gene expression dynamics using deep neural network
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- geneRNIB: a living benchmark for gene regulatory network inference
- Learning to Discover Regulatory Elements for Gene Expression Prediction
- A Variational Inference Approach to Single-Cell Gene Regulatory Network Inference using Probabilistic Matrix Factorization
- Information-incorporated gene network construction with FDR control
- Kolmogorov-Arnold Network for Gene Regulatory Network Inference
- Integrated Module and Gene-Specific Regulatory Inference Implicates Upstream Signaling Networks