Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

Auguste Schulz, Julius Vetter, Richard Gao, Daniel Morales, Victor Lobato Ríos, Pavan P Ramdya, Pedro J. Gonçalves, Jakob H. Macke

Abstract

Extracting the relationship between high-dimensional recordings of neural activity and complex behavior is a ubiquitous problem in systems neuroscience. Toward this goal, encoding and decoding models attempt to infer the conditional distribution of neural activity given behavior and vice versa, while dimensionality reduction techniques aim to extract interpretable low-dimensional representations. Variational autoencoders (VAEs) are flexible deep-learning models commonly used to infer low-dimensional embeddings of neural or behavioral data. However, it is challenging for VAEs to accurately model arbitrary conditional distributions, such as those encountered in neural encoding and decoding, and even more so simultaneously. Here, we present a VAE-based approach for accurately calculating such conditional distributions. We validate our approach on a task with known ground truth and demonstrate the applicability to high-dimensional behavioral time series by retrieving the conditional distributions over masked body parts of walking flies. Finally, we probabilistically decode motor trajectories from neural population activity in a monkey reach task and query the same VAE for the encoding distribution of neural activity given behavior. Our approach provides a unifying perspective on joint dimensionality reduction and learning conditional distributions of neural and behavioral data, which will allow for scaling common analyses in neuroscience to today's high-dimensional multi-modal datasets.

Introduction: The Challenge of Multi-Modal Data

In modern systems neuroscience, researchers are often tasked with interpreting relationships between high-dimensional neural recordings and complex behavioral outputs. Traditionally, this involves three distinct analytical pillars: dimensionality reduction (compressing data into latent variables), neural encoding (predicting neural activity from behavior), and behavioral decoding (predicting behavior from neural activity).

Conceptual goals of joint dimensionality reduction and conditional modeling in neuroscience.

While Variational Autoencoders (VAEs) have become a standard tool for dimensionality reduction, they typically struggle to perform encoding and decoding simultaneously. A standard VAE is trained to reconstruct its entire input; when presented with partial data (e.g., neural activity without behavior), it often fails to account for the resulting uncertainty. This leads to the "confidently wrong" problem, where a model produces biased estimates with unrealistically small error bars because it wasn't trained to handle the uncertainty of partial observations.

The Masked VAE Framework

To address this, we introduce a "structured masking" approach. By strategically hiding portions of the data during training, the VAE learns to produce accurate, well-calibrated conditional distributions regardless of which data modality is missing.

Overview of the masked VAE framework, including structured masking and latent dynamics with uncertainty.

Key Methodological Innovations

  1. Structured Masking: During training, the model is exposed to various combinations of masked data—sometimes hiding neural activity, sometimes behavior, and sometimes both. This forces the encoder to learn a posterior distribution that reflects the uncertainty of missing observations.
  2. Loss Function Adaptation: The reconstruction loss is computed only on the observed (non-masked) data. To handle different data types, we utilize Gaussian Negative Log-Likelihood (GNLL) for continuous behavioral data and Poisson distributions for neural spike counts, allowing the model to explicitly learn the noise level of each channel.
  3. Uncertainty Calibration: A critical component of the framework is ensuring that the model's uncertainty estimates are reliable. We employ an adaptation of simulation-based calibration to verify that the ground truth falls within the model's predicted confidence intervals at the expected frequency.

Experimental Validation

We validated the masked VAE across three distinct regimes:

Synthetic Ground Truth

Using a Gaussian Latent Variable Model (GLVM), we demonstrated that the masked VAE recovers the exact analytical posterior. This confirms that the masking strategy allows the model to approximate the true underlying mathematical relationship between variables.

High-Dimensional Fly Behavior

We applied the model to a 64-dimensional dataset of 32 tracked body parts from Drosophila melanogaster. The task was to predict the trajectories of hidden limbs based on the movement of visible ones. While a naive VAE could capture periodic rhythms, only the masked VAE correctly quantified uncertainty during non-periodic movements, providing a more realistic representation of the fly's motor variability.

Monkey Motor Cortex Decoding

In a classic monkey reach task, we decoded 2D cursor trajectories from the activity of over 200 neurons in the primary motor cortex. The masked VAE not only outperformed standard approaches in log-likelihood but also allowed us to query the same model for the encoding distribution (neural activity given behavior). This unification eliminates the need for separate encoding and decoding pipelines.

Discussion and Future Directions

The masked VAE provides a principled way to quantify how much a specific neural population constrains a specific behavior. By providing well-calibrated uncertainty estimates, it moves beyond simple point predictions to a more probabilistic understanding of brain-behavior relationships.

Future work may involve scaling this approach using Transformer-based architectures to capture longer temporal dependencies. Additionally, while the current model uses pre-specified structured masks, developing methods to handle arbitrary, unstructured missingness could further enhance data imputation capabilities in large-scale neuroscience datasets.

Summary updated

Abstract

Extracting the relationship between high-dimensional recordings of neural activity and complex behavior is a ubiquitous problem in systems neuroscience. Toward this goal, encoding and decoding models attempt to infer the conditional distribution of neural activity given behavior and vice versa, while dimensionality reduction techniques aim to extract interpretable low-dimensional representations. Variational autoencoders (VAEs) are flexible deep-learning models commonly used to infer low-dimensional embeddings of neural or behavioral data. However, it is challenging for VAEs to accurately model arbitrary conditional distributions, such as those encountered in neural encoding and decoding, and even more so simultaneously. Here, we present a VAE-based approach for accurately calculating such conditional distributions. We validate our approach on a task with known ground truth and demonstrate the applicability to high-dimensional behavioral time series by retrieving the conditional distributions over masked body parts of walking flies. Finally, we probabilistically decode motor trajectories from neural population activity in a monkey reach task and query the same VAE for the encoding distribution of neural activity given behavior. Our approach provides a unifying perspective on joint dimensionality reduction and learning conditional distributions of neural and behavioral data, which will allow for scaling common analyses in neuroscience to today's high-dimensional multi-modal datasets.
BibTeX
@inproceedings{schulz2025modeling,
  author = {Auguste Schulz and Julius Vetter and Richard Gao and Daniel Morales and Victor Lobato Ríos and Pavan P Ramdya and Pedro J. Gonçalves and Jakob H. Macke},
  title = {{Modeling conditional distributions of neural and behavioral data with masked variational autoencoders}},
  booktitle = {Cell},
  year = {2025},
  doi = {10.1016/j.celrep.2025.115338},
}
Figure