Deep probabilistic model synthesis enables unified modeling of whole-brain neural activity across individual subjects

William E. Bishop, Luuk Willem Hesselink, Bernhard Englitz, Misha B. Ahrens, James E. Fitzgerald

TL;DR

Researchers have developed Deep Probabilistic Model Synthesis (DPMS), a machine learning framework that allows data from unique individual subjects (like different brains) to be combined into a single, unified model. By using auxiliary metadata—such as the 3D location of a neuron—to predict model parameters, DPMS can "synthesize" insights across individuals even when their internal structures don't perfectly align.

What Problem Does This Paper Address?

In fields like neuroscience, economics, and ecology, researchers face a "many instances" problem. Every larval zebrafish brain has a different number of neurons in slightly different places. Because there isn't a 1-to-1 correspondence between "Neuron A" in Fish 1 and "Neuron A" in Fish 2, standard machine learning models are usually fit to one animal at a time.

This "N=1" approach is a major bottleneck. It limits generalizability (we learn about that fish, not the fish) and prevents us from pooling data to see the "big picture." If Fish 1 only performs "left turns" and Fish 2 only performs "right turns," a standard model can never learn a complete representation of turning behavior. This paper introduces a way to bridge these gaps by leveraging "auxiliary properties"—metadata that describes the system's components.

Methods

The core innovation is the Conditional Prior Distribution (CPD).

Instead of treating model parameters (like the weights of a neural network) as independent variables to be learned from scratch for every subject, DPMS assumes these parameters are predictable based on auxiliary properties ($M^s$). In the zebrafish example, the auxiliary property is the 3D spatial coordinate of each neuron.

  1. The CPD: A deep neural network learns a mapping from these spatial coordinates to a distribution of likely model parameters. This acts as a "biologically informed prior" shared across all subjects.
  2. Variational Inference: The framework uses variational inference to simultaneously optimize the shared CPD and subject-specific "posteriors."
  3. Synthesis: Because the CPD is shared, the model can "transfer" knowledge. If the model learns that neurons in a specific brain region are vital for swimming in Fish 1-4, it can apply that "prior" knowledge to Fish 5, even if Fish 5 has very little data.

The authors also utilize a specific functional form called Sum of Hyperrectangular Basis Functions (SHBF) for the CPD, which allows the model to handle non-smooth, complex relationships in low-dimensional property spaces efficiently.

Datasets

The authors validated DPMS using two primary sources:

  • Synthetic Data: 100 simulated "brains" where the ground-truth relationship between neural activity and behavior was known but varied across individuals.
  • Whole-Brain Zebrafish Data: Publicly available calcium imaging data from larval zebrafish (~80,000 neurons per fish). This included recordings of "fictive swimming" (motor nerve activity) while the fish responded to visual stimuli (phototaxis and optomotor responses).

Key Insights and Ideas

  • Metadata as a Bridge: The "aha moment" here is that while neurons themselves aren't identical across subjects, their properties (like location) follow general rules. By modeling the relationship between location and function, we can finally pool data from different individuals.
  • Alignment through Synthesis: Traditionally, if you want to compare the "latent states" (internal thoughts/dynamics) of two different brains, you have to perform complex post-hoc mathematical rotations to align them. DPMS does this automatically. Because all fish are tied to the same shared prior, their internal representations are "born" aligned.
  • Learning from "Missing" Data: DPMS can reconstruct a subject's behavior even in conditions that the subject never actually experienced. By seeing how other fish behave in "Condition A," the model uses the shared CPD to predict how the target fish would have behaved.

Key Results

  • Superior Decoding: In regression tasks (predicting swimming from neural activity), DPMS consistently outperformed models trained on single fish. This was especially true for "target" fish with limited training data; DPMS effectively "borrowed" strength from the "base" fish.
  • Unified Latent Spaces: DPMS successfully placed neural activity from different fish performing different behaviors (forward, left, and right turns) into a single, coherent 10-dimensional latent space.
  • Failure of Standard Methods: The authors showed that standard alignment techniques (like orthonormalization or distribution alignment) failed to correctly relate neural activity across fish when those fish were performing different behaviors. DPMS succeeded because it used the spatial metadata to anchor the models.

Limitations and Future Work

  • Low-Dimensional Properties: This study focused on 3D spatial coordinates as the primary auxiliary property. Future work could incorporate higher-dimensional data, such as genetic profiles or cell-type identities, which might provide even richer "priors" for model synthesis.
  • Factorization Assumptions: The current implementation assumes a "factorized" CPD (treating parameters as conditionally independent). While this simplifies the math and works well for many models (like linear regression or factor analysis), more complex neural architectures might require modeling correlations between parameters.
  • Scaling to Other Fields: While demonstrated in neuroscience, the authors note that DPMS is a general framework. It could theoretically be applied to any complex system with multiple instances, such as modeling different international financial markets or diverse ecological ecosystems.

Summary updated

Abstract

Many disciplines need quantitative models that synthesize experimental data across multiple instances of the same general system. For example, neuroscientists must combine data from the brains of many individual animals to understand the species' brain in general. However, typical machine learning models treat one system instance at a time. Here we introduce a machine learning framework, deep probabilistic model synthesis (DPMS), that leverages system properties auxiliary to the model to combine data across system instances. DPMS specifically uses variational inference to learn a conditional prior distribution and instance-specific posterior distributions over model parameters that respectively tie together the system instances and capture their unique structure. DPMS can synthesize a wide variety of model classes, such as those for regression, classification, and dimensionality reduction, and we demonstrate its ability to improve upon single-instance models on synthetic data and whole-brain neural activity data from larval zebrafish.
BibTeX
@misc{bishop2026deep,
  author = {William E. Bishop and Luuk Willem Hesselink and Bernhard Englitz and Misha B. Ahrens and James E. Fitzgerald},
  title = {{Deep probabilistic model synthesis enables unified modeling of whole-brain neural activity across individual subjects}},
  year = {2026},
  eprint = {2603.14161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.14161},
}

Loading related content...

Figure