Small-Cohort Calcium Biomarkers That Survive Cell Turnover

Chronic two-photon calcium imaging lets a lab watch the same patch of cortex for weeks while an animal learns, ages, or is treated. Labs increasingly want to turn these recordings into biomarkers: measurable properties of a neural circuit that differ between treated and untreated animals, or that change as a disease progresses. Two facts make this hard. First, studies use few animals, often four to ten. Second, the neurons themselves do not stay put. In one study only about 22% of neurons active on day 1 were still active on day 46 (EMC²). In hippocampus, only 16–17% of inhibitory interneurons could be re-identified across consecutive days, compared with 46–47% of excitatory cells (GABAergic CA1 neurons are more stable…). Standard small-sample biomarker pipelines expect every subject to have the same feature table. Chronic calcium data breaks that expectation, and breaks it unevenly across cell types and animals.

We propose to measure, rather than assume, which kinds of longitudinal calcium biomarkers replicate when discovered in a small cohort. We will build a replication benchmark from the Allen Brain Observatory two-photon data. Its 256 mice (POYO+) allow us to run thousands of simulated "small studies" and then check each discovery against held-out animals. We will plant known effects in some simulations and known confounds in others. Against this benchmark we will test one specific candidate method: an animal-level hierarchical model that treats each session's neurons as a sample from that animal's distribution and corrects for behavioral state. In this model, matched neurons add statistical strength, but no neuron has to be matched for the animal to count. The headline deliverable is a validated estimate, computed inside a small study, of how likely its finding is to replicate.

Core Claim

Our claim is that how a biomarker is defined matters more than how powerful the classifier is. Features defined on individual tracked neurons will replicate poorly under realistic turnover. So will features that ignore behavioral state, especially when animals differ in how much they run or how aroused they are. Features defined as animal-level distributional statistics will replicate better. Examples include quantiles of stimulus selectivity across a session's neurons, or the spread of variance across the dimensions of population activity. These should replicate better still when estimated with partial pooling across matched and unmatched neurons, after removing behavioral axes. We further claim that the usual in-study validation, cross-validation that splits trials or neurons rather than animals, overstates replication, and that an animal-level resampling estimate can be calibrated against held-out cohorts.

The closest existing work does something different. The BCI replication study (Validating neurophysiological predictors of BCI performance) re-tested six EEG predictors on 85 users and found that four replicated while the PPfactor and resting beta power did not. It audits predictors after the fact. It does not explain why they fail, and it does not address cellular data or longitudinal data. Across-animal alignment methods pool animals but do not validate group effects. amLDS (Across-animal odor decoding) decodes a new animal from two trials per stimulus, and CCA alignment (Preserved neural dynamics across animals) shows shared dynamics across animals. Both were evaluated within single sessions, and both assume that differences between animals are only differences in how a shared signal is projected onto each animal's neurons. Work on behavioral variability (Contribution of behavioural variability to representational drift) motivates our behavioral correction: it showed that pupil size and running move population activity along their own dimensions. That work stopped at within-day recordings and explicitly left multi-day timescales open.

Research Program

The benchmark has three layers.

The first layer contains real-label tasks. Each simulated study draws N of 4, 6, 8, or 10 mice from two Cre lines or two areas. It discovers features that separate the groups, then tests them on the remaining mice. Effects in these tasks are real and their size is known from the full cohort.

The second layer adds planted longitudinal effects. Within one Cre line, we randomly label half the sampled mice "treated". In their later sessions we scale stimulus-evoked responses or pairwise correlations by a known factor, in a known fraction of neurons. We then impose cell turnover that matches the published retention curves: about 22% retention by the final session for excitatory-like populations, following EMC², and 16–17% session-to-session for interneuron-like populations, following the CA1 study.

The third layer contains planted nulls. Group labels are assigned according to each animal's mean running speed, and no neural effect is planted. Any neural "biomarker" discovered here is a false positive driven by behavior.

The candidate method models a per-neuron feature $f_{msi}$, for animal $m$, session $s$, and neuron $i$, as

$$f_{msi} = \mu_m + \delta\, g_m t_s + \beta^\top b_{ms} + u_{i} + \varepsilon_{msi}.$$

Here $\mu_m$ is the animal's baseline level. The term $g_m t_s$ is the group label times session time, so $\delta$ is the longitudinal group effect being tested. The vector $b_{ms}$ holds behavioral covariates for that session, such as running speed and pupil where available, with coefficients $\beta$. The term $u_i$ is a neuron random effect. It is shared across sessions only when registration matched neuron $i$, so unmatched neurons still inform $\mu_m$ and $\delta$. Inference on $\delta$ uses the number of animals as its sample size, not the number of neurons. This model says nothing about why a neuron disappeared, whether through death, silence, or drift out of the imaging plane. That limitation is deliberate. We are betting that a biomarker robust to turnover need not resolve the "blinking" problem, in which a quiet neuron emits almost no fluorescence and looks absent. The planted-turnover layer tests that bet directly.

Features will be computed from both raw fluorescence change (ΔF/F) and inferred spike rates. Spike inference has a known accuracy ceiling: the best methods reach a correlation of about 0.44 with electrophysiological ground truth (spikefinder). A biomarker whose replication depends on which spike-inference method was used should therefore be distrusted.

The in-study replication estimate is computed by leave-animals-out resampling of the whole discovery pipeline, including feature selection. The resampling distribution of $\delta$ is converted into a predicted probability that the effect is significant, in the same direction, in an independent cohort of the same size.

Baselines

The naive reference is the classical summary-statistic approach. For each animal and session, compute the population mean of a feature, take the change across sessions, and run a two-sample test across animals. It uses all neurons without any matching, it is what most labs would do, and it is the method our approach most needs to beat. It will run on identical simulated studies. Our method beats it if it achieves at least the same replication rate at a lower false-positive rate on the behavior-confounded nulls, or a higher replication rate on planted effects of small size.

The second baseline is the high-dimensional pipeline used in small-sample omics, as in the APOC1 study (APOC1 as a novel diagnostic biomarker). That study used penalized regression that selects a few features (LASSO) and support-vector-machine recursive feature elimination (SVM-RFE). It reported an AUC of 92.5% on an independent 20-versus-20 cohort. We will apply the same pipeline to tables of matched-neuron features, restricted to neurons that are present in every session. This represents the strongest form of the "consistent feature table" approach.

The third baseline is foundation-model features. We will pretrain POYO+ on the mice excluded from each simulated study, then fit a linear probe at the animal level on pooled session embeddings. POYO+ reports nearly a 10% gain over single-session MLP and temporal-convolution baselines on orientation decoding. It also reports a hippocampal position error of 9.58 cm, versus 10.33 cm when trained from scratch. No reported result found on Lacuna for POYO+ on animal-level group discrimination or replication.

The fourth baseline is amLDS-style alignment followed by a group test on the aligned latent states. amLDS reports about 82% across-animal odor decoding accuracy.

All methods get the same studies, the same held-out animals, and the same budget for tuning hyperparameters inside the discovery set. No method may see held-out animals during model selection.

Two ablations isolate our method's ingredients. The first drops the behavioral term $\beta^\top b_{ms}$. The second drops partial pooling of unmatched neurons and uses matched neurons only.

Why not just…?

Why not just average each animal's neurons and run a t-test? This is the real threat. In many settings it may perform as well as our model. Our hierarchical model gains only where turnover differs between groups or where behavior differs between animals. If the averaging baseline matches ours on both planted effects and behavioral nulls, we will report that, and recommend averaging combined with our calibrated resampling estimate. The calibration result would still be useful without the model.

Why not just keep the neurons that are tracked in every session? Doing so discards about 78% of units over 46 days and almost all interneurons over a few days, following EMC² and the CA1 study. It also selects for unusually stable cells, which biases any estimate of change across sessions. The matched-only ablation and the LASSO baseline measure the cost directly.

Why not just use a pretrained foundation model? POYO+ was built for decoding behavioral or stimulus variables from many healthy mice. Its embeddings may encode exactly the behavioral axes that create false positives. Its performance on the running-speed nulls will settle this question. If it replicates well and resists the confound, the practical recommendation becomes "probe POYO+ embeddings at the animal level".

Why not just regress out running speed? Regressing out running speed is one ingredient of our model, not a competitor to it. The behavior ablation shows how much of the gain it accounts for on its own.

Proposed Experiments and Decisive Checks

The primary metrics are four. The first is the replication rate: the fraction of small-study discoveries that are significant, in the same direction, in held-out mice. The second is the false-positive rate on planted nulls. The third is effect-size inflation, the ratio of the discovered effect to the held-out effect, which measures the "winner's curse". The fourth is calibration of the in-study replication probability against observed replication, measured as reliability across bins of predicted probability.

The decisive checks are three. First, on planted nulls, any method without the behavioral term should show elevated false positives, and the hierarchical model should not. Second, with planted effects under interneuron-like turnover, matched-only pipelines should lose power faster than distributional features as N falls from 10 to 4. Third, the leave-animals-out replication estimate should be calibrated, while trial-split cross-validation should be measurably overconfident. If the third check fails, the central methodological claim fails with it.