Letting a Cortical Foundation Model Correct Its Own Atlas Registration

Models that decode behavior from many animals' brains first warp each animal's brain onto a shared anatomical map, an atlas, so that "pixel 412" means the same cortical spot in every mouse. This warping, called spatial normalization, is done once, before training, and never revisited. Any misregistration is permanently baked into the model's inputs. We propose to close the loop. The trained model's own internal representations are used to find cortical locations that behave alike across mice. Only the most trustworthy matches, those that agree in both directions and exceed a confidence threshold, are kept. Each mouse's warp is re-fitted from these matches, the data are re-cut into tokens, and training continues. This repeats the self-learning recipe that made cross-lingual word-embedding alignment robust, with spatial warps standing in for rotations of an embedding space.

The setting is widefield calcium imaging of mouse dorsal cortex, where WiCAT (Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization) showed that atlas alignment is what makes zero-shot decoding on unseen mice possible. If alignment matters that much, then errors in alignment should cost accuracy. A loop that removes those errors without any labels, including on a new mouse at test time, should raise cross-subject decoding and tighten functional maps across animals.

Core Claim

The central claim is that a functional representation, learned from many animals and then used to bootstrap a set of mutually verified correspondences, can refine per-animal atlas registration better than two alternatives. The first is classical image re-registration. The second is a learnable warp trained end-to-end through the model's reconstruction loss. The specific ingredients carried over from the alignment literature are four. The existing anatomical registration serves as a weak seed rather than starting from scratch. Correspondences are harvested by mutual nearest neighbors in the model's embedding space. Each harvested match is weighted by a continuous reliability score instead of being trusted outright. Representation and registration are updated alternately, so each improves the other.

Where This Sits in Prior Work

The bootstrapping side supplies the method. Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? breaks embedding alignment into seed, self-learning loop, and normalization. It shows that the loop, started from a small trusted seed, does most of the work: weakly supervised alignment averaged 0.274 Mean Reciprocal Rank against 0.140 for fully unsupervised, and the unsupervised version collapsed on 41.4% of language pairs. This finding also constrains our design. We do not attempt seed-free registration. The standard landmark-based atlas fit is our seed. Three other alignment papers supply the error controls. How to (Properly) Evaluate Cross-Lingual Word Embeddings analyzes mutual-nearest-neighbor matching as a guard against "hub" points that are close to everything. Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision adds hard similarity thresholds on top of it. Iterative Entity Alignment via Joint Knowledge Embeddings shows that weighting pseudo-matches by reliability ("soft alignment") beats treating them as ground truth, reaching 71.7% Hits@1 against 38.9% for a non-iterative baseline. Self-Supervised Neural Machine Translation motivates the alternating design: retrieval improves representations, which improve retrieval. Our contribution is to move all of this from abstract vector spaces onto a physical 2D cortical sheet. There, the "transformation" is a smooth deformation rather than an orthogonal matrix.

On the normalization side, WiCAT and Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models motivate the gap. Both pool subjects through a static atlas or cortical surface projection, and the latter cuts per-subject data needs fivefold this way. Neither lets the learned model revise that projection. VoteNet++ is the closest in spirit: it iteratively refines a dense displacement field under a bending-energy smoothness penalty, guided by a frozen segmentation network. It never harvests discrete verified correspondences, and its guiding network is never updated by the refined registration. We borrow its smoothness penalty and add both missing pieces. Few-shot image segmentation for cross-institution male pelvic organs predicts alignment inside feature space, but only as a one-shot affine map. Best-Buddies Similarity shows that mutual nearest neighbors are robust to clutter and deformation in image matching, which supports using mutual verification spatially, though it has no loop. Finally, landmark-driven pipelines such as BrainsMapi, which needs 4–6 hours of manual feature delineation per brain, and the vascular-fiducial registration of Accolla et al. show what our loop replaces: hand-chosen anchors that never grow.

Research Program

We first reproduce WiCAT-style training on the Musall (13 mice, 26 sessions) and Kondo (25 mice, 352 sessions) datasets. The model is a masked-autoencoding transformer over atlas-aligned 32×32 spatiotemporal patches, using each dataset's standard atlas registration. This trained model is iteration zero.

Each refinement iteration has four steps. The first step computes an embedding for every cortical pixel of every mouse, read from the encoder's token outputs and interpolated within patches. The second step matches each mouse to a population template, the running mean embedding map in atlas coordinates. We keep pairs that are mutual nearest neighbors within a bounded search radius and exceed a similarity threshold. Each kept pair receives a weight $w_i$, its similarity margin over the second-best match. The third step fits a smooth per-mouse deformation $\phi$ by minimizing

$$\sum_i w_i \,\lVert \phi(x_i) - y_i \rVert^2 + \lambda\, E_{\text{bend}}(\phi),$$

where $x_i$ is a location in the mouse's current registration, $y_i$ its matched template location, $E_{\text{bend}}$ the bending energy of $\phi$, and $\lambda$ the smoothness weight. The fourth step re-warps the raw frames, re-tokenizes them, and fine-tunes the encoder for a fixed budget before the next iteration.

Regions where no confident mutual match exists, such as window edges, vessel shadows, or genuinely idiosyncratic areas, receive no anchors. The smoothness term carries the seed warp through them. This is the spatial analogue of refusing to align non-isomorphic parts of two embedding spaces.

For an unseen mouse, the loop runs on its neural data alone, with the encoder frozen, before decoding. No behavioral labels are used, so zero-shot evaluation stays valid. We also track how the anchor set grows and how stable the warps are across iterations. This tests whether confirmation bias appears, in which a wrong early warp recruits more wrong matches.

Baselines

The naive reference is static registration: WiCAT trained once on the standard atlas alignment, with no refinement. WiCAT reports zero-shot behavior-decoding R² of 0.327 on Musall and 0.184 on Kondo (Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization). We will report both these figures and our own reproduction, and all comparisons will use the reproduction so the code and splits match.

The strongest competitor is a learnable per-mouse deformation, with the same parameterization and bending penalty as ours, optimized by backpropagating the masked-reconstruction loss through a differentiable resampler, jointly with the encoder. It is the strongest competitor because it targets exactly what our loop targets: registration that serves the model, with no labels. It gets the same data, the same total GPU hours, and the same hyperparameter search budget over $\lambda$ and learning rates. Hyperparameters are chosen on held-out validation mice for both methods. At test time it adapts its warp on the unseen mouse's neural data, just as our method does. No reported result found on Lacuna for this baseline.

The classical competitor re-registers each mouse's per-pixel functional summary map to the population template with a standard intensity-based deformable registration tool such as NiftyReg (one of the classical tools benchmarked in Learn2Reg 2024). It uses the same smoothness budget and then retrains the model once. The functional summary is each pixel's correlation profile to atlas areas. No reported result found on Lacuna for this on widefield data.

A spatial-jitter baseline trains WiCAT with random smooth warps as data augmentation, so the model becomes tolerant of misregistration instead of correcting it.

Ablations isolate the key ingredients. We replace mutual matching with one-way nearest neighbors. We replace soft weights with hard acceptance, mirroring the hard-versus-soft comparison of Zhu et al. We run a single refinement pass with frozen features, which keeps registration correction but removes the alternating loop. We keep the loop but never fine-tune the encoder, which removes the two-way feedback.

Beating the field means a statistically significant gain in held-out-mouse R² on both datasets over the end-to-end warp, under matched compute and tuning. It also requires no worse deformation smoothness, measured by the standard deviation of the log Jacobian determinant as in Learn2Reg.

Why not just…?

Why not just learn the warp end-to-end through the reconstruction loss? This is the real threat, and the head-to-head comparison above settles it. Our reason to expect it to fall short is specific. A reconstruction objective rewards warps that make the input easier to predict, for example by smearing or compressing high-variance regions, and nothing ties the warp to correspondence across animals. The Vulić et al. result that purely self-supervised alignment objectives collapse on a large fraction of cases points the same way. The synthetic-warp experiment below measures this directly: if the end-to-end warp recovers injected displacements as well as our loop, we will report that the bootstrapping machinery is unnecessary.

Why not just re-register functional maps with a classical tool? Classical methods win low-data, cross-modal registration in Learn2Reg 2024, so this is credible. However, they match intensities of a hand-chosen summary image and lack a mutual-consistency test, so idiosyncratic regions pull the warp. The classical baseline and the one-way-neighbor ablation together show whether mutual verification over learned features is what helps.

Why not just augment with spatial jitter and let the model absorb misalignment? Tolerance to noise is not the same as correction. A model that blurs over misregistration must lose spatial resolution, and WiCAT and Moussa & Toneva both attribute their gains to precise shared coordinates. If jitter matches our R², that indicates decoding at this patch size is insensitive to registration. The positive-control experiment below would detect this first.

Proposed Experiments

The first and decisive check is a positive control. We inject known smooth warps of increasing magnitude into the atlas registration of training and test mice and measure how much WiCAT's zero-shot R² drops. If decoding barely moves at plausible misregistration scales, the premise is wrong and we stop, reporting that 32×32 tokenization is coarse enough to make fine registration irrelevant.

If decoding is sensitive, the same injected warps give ground truth. We measure displacement-recovery error in micrometers for our loop, the end-to-end warp, and the classical re-registration. We also measure R² recovery toward the uncorrupted model.

On the real data, the primary outcome is zero-shot R² on held-out mice for Musall and Kondo across multiple splits. Secondary outcomes are few-shot adaptation curves against WiCAT's 16-trial result, and cross-mouse spread in the locations of sensory-evoked activation peaks, a label-light test of registration quality. Smoothness is reported throughout as the standard deviation of the log Jacobian determinant.

Iteration traces track anchor count, the fraction of anchors later revoked, and R² per round. A loop that grows anchors while revocation rises, and R² falls, is the confirmation-bias failure. The hard-acceptance ablation should exhibit it, and the soft, mutual version should not. If time permits, the same loop applies unchanged to Moussa & Toneva's cortical-surface fMRI pooling, tested by brain-alignment accuracy on held-out Narratives participants.