Two research communities are looking at the same phenomenon from opposite sides and missing each other. Human pharmacological imaging studies have shown, twice and independently, that oxytocin's effects on the brain are conditional on serotonin: intranasal oxytocin switches the left amygdala from sensitizing to habituating under repeated threat, and acute tryptophan depletion (a drink that temporarily starves the brain of serotonin's precursor) largely cancels that switch (Oxytocinergic Modulation of Threat-Specific Amygdala Sensitization in Humans Is Critically Mediated by Serotonergic Mechanisms); the same holds for oxytocin's strengthening of amygdala–hippocampus connectivity (Oxytocinergic Modulation of Stress-Associated Amygdala-Hippocampus Pathways in Humans Is Mediated by Serotonergic Mechanisms). Meanwhile, the functional-enrichment community builds statistical tools that decide which biological pathways are activated in a gene list — but every tool in the current arsenal, from Fisher's exact test up to regularized regression, tests pathways one condition at a time and assumes effects combine linearly (Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression). The gating phenomenon the neuroscientists keep finding is exactly the non-linear, conditional structure the enrichment tools cannot express.
We propose to close this gap from both ends at once. First, we will generate the missing transcriptomic data: a full 2×2 factorial perturbation of the oxytocin and serotonin systems (oxytocin vs. vehicle crossed with serotonin depletion vs. control), profiled at single-nucleus resolution in the hypothalamus, amygdala, and medial prefrontal cortex, in both sexes — the factorial design the imaging studies used but no sequencing study has. Second, we will build a gated enrichment method that explicitly tests, per pathway, whether its activation under oxytocin depends on the serotonin state, by extending the regularized-regression enrichment framework with interaction terms and a factorial contrast, validated on synthetic benchmarks with planted gating ground truth. The deliverable is both a dataset and a statistical tool, each useless without the other.
Core Claim
The claim is that serotonin's mediation of oxytocin's effects is not only a circuit-level phenomenon but a molecular-pathway-level phenomenon that is statistically detectable and enumerable: there exists a specific, small set of pathways whose enrichment score under oxytocin treatment flips depending on serotonin availability, and current enrichment methods — which test each condition separately and intersect the results — cannot distinguish a gated pathway from one that shows additive effects in both arms. We formalize gating as the factorial interaction contrast. For a pathway $A$, let $s_A(\text{OXT}, \text{5HT})$ be its activity score in the oxytocin/serotonin cell of the design. Then the gating statistic is
$$G_A = \big[s_A(\text{OXT}, 5\text{HT}^+) - s_A(\text{PLC}, 5\text{HT}^+)\big] - \big[s_A(\text{OXT}, 5\text{HT}^-) - s_A(\text{PLC}, 5\text{HT}^-)\big],$$
where $5\text{HT}^+$ and $5\text{HT}^-$ denote intact and depleted serotonin, and PLC is placebo. A pathway is serotonin-gated when $G_A$ is large relative to a permutation null that shuffles serotonin-arm labels within matched samples, which preserves each marginal drug effect while destroying the interaction. This contrast is exactly what a per-arm analysis cannot compute: two pathways with identical $G_A$-free additive responses can produce identical per-arm significance lists yet completely different gated behavior.
This is deliberately different from the closest existing work in three ways. The regularized-regression enrichment method pareg models how pathway memberships linearly combine to predict gene-level differential-expression p-values, with a network-fusion penalty that captures between-pathway similarity (Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression) — its authors flag their own linearity assumption as a major limitation, and we remove it by adding pathway-membership × condition interaction terms to their model rather than inventing a new one. The autism review that is the only existing paper at this intersection (Oxytocin and serotonin in the modulation of neural function: Neurobiological underpinnings of autism-related behavior) argues conceptually that OXT and 5-HT act as parallel modulators on overlapping circuits and calls for single-cell sequencing and organoids to open the "black box" between genes and behavior — but it performs no enrichment analysis, no transcriptomic profiling, and crucially treats the two systems as parallel rather than interacting, which is the relationship the imaging data actually show. Finally, embedding- and text-based tools like GSPA and gtCLIP improve which pathways get flagged and how redundant hits get grouped (Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings; Contrastive Learning for Gene Set Enrichment Analysis Post-Processing) — they are orthogonal to gating and we reuse their machinery where noted below rather than compete with it.
Research Program
The program has three components. The first is data generation. We will use a rodent model carrying the perturbation tools the field already trusts: systemic or intranasal oxytocin vs. vehicle, crossed with serotonin depletion (the animal analog of acute tryptophan depletion, plus a 5-HT receptor blockade arm), delivered in the same 2×2 between-subject design as the human studies. We profile single-nucleus transcriptomes of the paraventricular hypothalamic nucleus, amygdala (with basolateral and central subnuclei separated, resolving the anatomical coarseness the human studies acknowledge), and medial prefrontal cortex — the three hubs the autism review identifies as the convergence sites of the two systems, and the circuit the sleep-deprivation work already connects mechanistically: high-frequency optogenetic stimulation of paraventricular oxytocin neurons drives oxytocin release in the medial prefrontal cortex and restores inhibitory transmission there, reversing anxiety-like behavior (The Oxytocin Neurons in the Paraventricular Nucleus Are Essential for Chronic Sleep Deprivation‐Mediated Anxiety‐Related Behaviors). We include both sexes throughout, since every existing human crosstalk study is male-only and the sexual dimorphism of both systems is a documented, unresolved limitation of all of them (Oxytocinergic Modulation of Threat-Specific Amygdala Sensitization in Humans Is Critically Mediated by Serotonergic Mechanisms; The oxytocin receptor rs2254298 polymorphism and alcohol withdrawal symptoms: a gene–environment interaction in mood disorders).
The second component is the method. We extend the pareg generalized linear model — which regresses gene-level differential-expression p-values on a binary pathway-membership matrix with a LASSO penalty for sparsity and a network-fusion penalty for pathway similarity (Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression) — by pooling all four arms and adding, for every pathway, membership × oxytocin, membership × serotonin-depletion, and the triple membership × oxytocin × serotonin-depletion interaction coefficient. The triple coefficient is the gene-level version of $G_A$; permuting serotonin-arm labels within animal-matched strata gives the null. The network-fusion penalty stays, which matters here because, as the gtCLIP analysis showed, over 80% of pathway pairs share no genes at all, so oxytocin- and serotonin-cascade pathways will be near-zero Jaccard neighbors in the penalty graph and their joint inference needs either the text-embedding similarity that gtCLIP introduced (Contrastive Learning for Gene Set Enrichment Analysis Post-Processing) or the geometric expansion that GSPA computes over protein-interaction networks (Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings). We use both as similarity sources for the fusion penalty.
The third component addresses a threat the file makes explicit: the knowledge bases these tools sit on are thin exactly where we need them. GSPA's authors found their network-embedding method performs noticeably worse on neurodegenerative than oncological phenotypes and attribute this to underrepresentation of neurobiological interactions in literature-derived networks (Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings), and the DeepSyn work documented the "discovery paradox" in which standard databases entirely miss context-specific function hierarchies (Deep functional synthesis: a machine learning approach to gene functional enrichment). We therefore audit the representation of the oxytocin–serotonin axis (OXTR, HTR1A/2A, and their second-messenger cascades) in the STRING, KEGG, and GO resources underlying our similarity matrices, and where coverage is thin we supplement the interaction network with literature mining in the spirit of DeepSyn's 16.7-million-abstract text pipeline — its knowledge graph of 85,000 entities and roughly 3 million associations already demonstrates that context-conditioned functional links can be recovered that static lookups miss entirely.
Baselines
The naive reference point is what a lab does today: run Fisher's exact test or standard GSEA separately on each arm's differentially expressed gene list, flag pathways significant in the oxytocin arm, and inspect which of those also appear in the serotonin-depleted arm. This "intersect the lists" practice is the default because it needs no new software, and it is the correct null hypothesis for our claim: if gating is a strong enough phenomenon, the intersection should partially recover it. The comparison is run on the same synthetic and real data with the same pathway universe, and we beat it only if we recover gated pathways at higher precision with calibrated interaction p-values, since the intersection carries no variance estimate for the contrast at all. The benchmark evidence that this baseline underperforms is indirect but consistent: in the regularized-regression benchmark suite, Fisher's exact test, topGO, MGSA, and blitzGSEA all trailed pareg, which achieved the highest mean Area Under the Precision-Recall Curve across noise levels (Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression); no reported result found on Lacuna for a numeric mean AUPRC value for pareg or Fisher's exact test in that benchmark, so our comparison must generate its own numbers on planted ground truth.
The strongest existing competitor is pareg itself, run per arm and then post-hoc compared: fit the regularized model in each of the four cells, compute each pathway's coefficient difference between the oxytocin and vehicle arms within each serotonin state, and call a pathway gated when the two differences disagree. This is the fair fight, because our method is pareg plus interaction terms — if joint modeling of the interaction buys nothing over per-arm fits stitched together afterward, the proposal's key ingredient is worthless. We run it with the same pathway sets, the same network-fusion penalty, the same cross-validation and tuning budget, and the same permutation machinery, so the only difference is whether the interaction is estimated jointly with the marginals or reconstructed post hoc. We beat it if, on planted synthetic data, jointly estimated gating coefficients achieve higher AUPRC for gated-pair recovery at matched false-discovery rate, and if the post-hoc procedure shows the failure mode we predict: it cannot distinguish a gated pathway from an additive one when both share a similar per-arm significance pattern. The ablation that isolates our key ingredient is our full model with the interaction terms deleted, which collapses back to a pareg-style linear model on pooled data — this must lose exactly the gated pairs and nothing else if the ingredient is doing what we claim — plus the interaction terms with the network-fusion penalty removed, which tests whether cross-pathway joint inference is necessary or the interaction alone suffices.
Why not just…?
Why not just run standard GSEA or Fisher's exact test on each arm and intersect the pathway lists? This is the naive baseline above, and it fails for two reasons the file documents directly. Statistically, the intersection has no test for the interaction: it conflates a pathway activated additively by both drugs with one whose oxytocin activation requires serotonin, and the synthetic benchmark showed Fisher-style discrete tests already trail dependency-aware regression even on single conditions (Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression). Structurally, convergent oxytocin and serotonin cascades may share almost no genes — over 80% of pathway pairs share none at all (Contrastive Learning for Gene Set Enrichment Analysis Post-Processing) — so an intersection on gene-set membership misses exactly the cross-system relationships this proposal targets. We will still run it as the reference point, and if it recovers the planted gated pairs as well as our method on the synthetic benchmark, that result would count against the proposal and we would report it.
Why not just cluster the significant pathways by Jaccard gene overlap afterward? Because the file shows overlap-based post-processing fragments precisely the relationships we care about: gtCLIP achieved 3.3-fold higher within-cluster gene set overlap and 3.9-fold higher silhouette scores than the strongest overlap-based clustering, precisely because most functionally related pairs share zero genes (Contrastive Learning for Gene Set Enrichment Analysis Post-Processing). We therefore use gtCLIP-style text-embedding similarity inside the fusion penalty rather than Jaccard, and one of our checks is whether this choice changes which pathways are flagged as gated.
Why not just look up the curated "Oxytocin signaling pathway" and "Serotonergic synapse" entries in KEGG? Static lookups answer membership, not conditionality, and the file gives three reasons to distrust them here: KEGG treats functions as context-free, while DeepSyn showed static tools miss active, context-specific hierarchies entirely and outperform static recovery by at least 18% (Deep functional synthesis: a machine learning approach to gene functional enrichment); network knowledge bases are documented as incomplete and non-cell-type-specific, with network-informed testing finding 25 significant pathways in TCGA breast cancer data where standard methods found none or almost none at FDR 0.01 (Network-based pathway enrichment analysis with incomplete network information); and neurobiological interactions are systematically underrepresented in the literature-derived networks these databases are built from (Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings). Our knowledge-base audit experiment addresses this head on and quantifies how much it matters for this specific axis.
Why not just use bulk RNA-seq instead of single-nucleus profiling? Bulk is cheaper and the pipelines are mature, but the gating signal we are hunting lives in small, specific cell populations — paraventricular oxytocin neurons and their prefrontal targets, a circuit where a defined neuronal subtype's AMPAR currents and peptide release carry the effect (The Oxytocin Neurons in the Paraventricular Nucleus Are Essential for Chronic Sleep Deprivation‐Mediated Anxiety‐Related Behaviors) — and bulk averaging dilutes subtype-confined signals. The single-cell enrichment evidence supports this: DeepGSEA showed higher sensitivity than ssGSEA, GSVA, AUCell, Vision, SCPA, and scGSEA across four simulation studies precisely when differential signals are subtle and confined to subpopulations (DeepGSEA: explainable deep gene set enrichment analysis for single-cell transcriptomic data). We will nevertheless run the analysis on pseudo-bulked profiles as an ablation; if gating survives bulk averaging, the cheaper design wins and we will say so.
Proposed Experiments
The first experiment is the synthetic benchmark, which settles the methodological question with ground truth. We extend the simulation design used for the pareg benchmark (differential-expression p-values simulated across noise levels and pathway similarity factors) and the Splatter-based simulations from DeepGSEA, planting three kinds of pathway behavior: purely additive (both drugs act independently), gated (oxytocin effect present only with serotonin intact — the pattern the human imaging found), and suppressed (effect present only when serotonin is depleted). Metric is AUPRC for recovering the gated set, plus the false rate at which each method labels additive pathways as gated — the error the intersection baseline is structurally guaranteed to make and the one that matters most scientifically. The same corpora reuse the established evaluation resources: the 12,612-gene-set corpus aggregated from GO, KEGG, Reactome, and WikiPathways (Contrastive Learning for Gene Set Enrichment Analysis Post-Processing) for the pathway universe, and the GEO2KEGG collection of 42 disease-matched expression profiles across 19 diseases (Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings) for testing whether adding interaction terms degrades ordinary single-condition enrichment, where our method must match pareg rather than beat it.
The second experiment applies the method to the new factorial single-nucleus dataset. The decisive check is directional: the model must flag GPCR and oxytocin-cascade pathways as serotonin-gated (the molecular readout of the imaging findings), and — more informatively — it must not flag pathways where the imaging data show oxytocin acting independently of serotonin. The per-cell-type resolution lets us test a concrete mechanistic prediction derived from the sleep-deprivation work: if restoring prefrontal inhibition is an oxytocin-dependent, circuit-specific effect (The Oxytocin Neurons in the Paraventricular Nucleus Are Essential for Chronic Sleep Deprivation‐Mediated Anxiety‐Related Behaviors), then GABA-synapse enrichment in prefrontal interneurons, but not in hypothalamic cells, should carry a significant gating coefficient. We also stratify by sex, since the genotype work shows oxytocin-receptor variation interacts with physiological stress to predict anxiety but not depression, with the interaction strongest in homozygous GG carriers (β = 0.23 for the interaction overall, slopes of 0.27 for A-allele carriers vs. 0.50 for GG homozygotes, in a weak differential-susceptibility model with R² = 0.17, in 265 men) (The oxytocin receptor rs2254298 polymorphism and alcohol withdrawal symptoms: a gene–environment interaction in mood disorders) — the transcriptomic gated set is the first place such genotype dependence could be examined at pathway level, and differing gated sets between sexes would directly quantify the generalizability limitation every source study acknowledges.
The third experiment is the knowledge-base audit with a functional endpoint: we measure how well the STRING/KEGG-derived similarity matrices used in our fusion penalty represent the oxytocin–serotonin axis, then test whether literature-augmented similarity changes the recovered gated set. The evaluation reuses the Alzheimer's disease corpus from the DeepGSEA evaluation set (DeepGSEA: explainable deep gene set enrichment analysis for single-cell transcriptomic data) as a known-hard neurological case where embedding-based enrichment underperforms, on the logic that if augmentation helps there it will help on the even less-represented neuroendocrine axis.
The closing check that would falsify the core claim: if, on planted synthetic data, the joint interaction model does not beat the per-arm pareg post-hoc comparison and the list-intersection baseline at recovering gated pathways, or if on real data the gated set is statistically indistinguishable from what additive per-arm analysis already finds, then serotonin's mediation of oxytocin effects is not enumerable at the pathway level, and the right level of description remains the circuit — a genuine, publishable answer either way, and one the field currently cannot give because neither the data nor the statistic exists.
References
- Oxytocinergic Modulation of Threat-Specific Amygdala Sensitization in Humans Is Critically Mediated by Serotonergic Mechanisms
- Oxytocinergic Modulation of Stress-Associated Amygdala-Hippocampus Pathways in Humans Is Mediated by Serotonergic Mechanisms
- Coherent pathway enrichment estimation by modeling inter-pathway dependencies using regularized regression
- Oxytocin and serotonin in the modulation of neural function: Neurobiological underpinnings of autism-related behavior
- Gene set proximity analysis: expanding gene set enrichment analysis through learned geometric embeddings
- Contrastive Learning for Gene Set Enrichment Analysis Post-Processing
- The Oxytocin Neurons in the Paraventricular Nucleus Are Essential for Chronic Sleep Deprivation‐Mediated Anxiety‐Related Behaviors
- The oxytocin receptor rs2254298 polymorphism and alcohol withdrawal symptoms: a gene–environment interaction in mood disorders
- Deep functional synthesis: a machine learning approach to gene functional enrichment
- Network-based pathway enrichment analysis with incomplete network information
- DeepGSEA: explainable deep gene set enrichment analysis for single-cell transcriptomic data