Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data Transformers

Linxing Preston Jiang, Shirui Chen, Emmanuel Tanumihardja, Xiaochuang Han, Weijia Shi, Eric Shea‐Brown, Rajesh P. N. Rao

TL;DR

This paper investigates how data heterogeneity in neuroscience datasets impacts the performance of pre-trained neural models. They found that while pre-training generally helps, significant heterogeneity (like different brain regions being recorded) can limit scaling benefits. Surprisingly, a small, carefully selected subset of sessions can outperform models trained on a much larger, randomly chosen set, suggesting that not all data is created equal for pre-training.

What Problem Does This Paper Address?

Analyzing neuroscience data is tough because it's inherently messy and variable. Think about recordings from different animals, different days (sessions), and even different parts of the brain – they all look and behave differently. This variability is called "heterogeneity." While pre-training large neural network models on massive datasets has been a game-changer in fields like language and vision, it's not clear if this "scaling law" effect holds true for neuroscience data. Specifically, researchers want to know:

  • How does the amount of pre-training data affect model performance, especially when dealing with this inherent data heterogeneity?
  • Do all pre-training sessions contribute equally, or are some more valuable than others?
  • Can we still get good performance gains from pre-training even with these challenges?

Methods

The researchers used a Neural Data Transformer (NDT) architecture, a type of model well-suited for sequential neural data. Their core methodology involved:

  • Systematic Scaling Analysis: Instead of just comparing a single-session model to a model trained on the entire dataset, they incrementally increased the number of pre-training sessions and observed how performance changed. This allowed them to see the scaling behavior.
  • Investigating Heterogeneity Sources: They specifically looked at two types of heterogeneity:
  • Explicit Heterogeneity: Differences in recorded brain regions across sessions. They used two datasets that varied in how consistently they recorded from the same regions.
  • Implicit Heterogeneity: Variability between sessions that isn't due to explicit differences like brain region, but rather subtle variations in animal behavior or recording conditions.
  • Session-Selection Procedure: To understand the impact of implicit heterogeneity, they developed a method to rank pre-training sessions. This involved finetuning a model on a held-out session using transformers pre-trained on each individual session, and then ranking sessions based on how well those transformers performed. This helped identify "better" sessions for pre-training.
  • Multi-Masking Scheme: For pre-training, they used a technique where parts of the input neural data were masked, and the model had to reconstruct them. This included masking based on co-smoothing (neurons), forward-prediction (time steps), inter-region (brain regions), and intra-region (neurons within a region).

Datasets

The study utilized two large-scale, multi-brain-region, multi-animal/session datasets from the International Brain Laboratory (IBL).

  • RepeatedSite (RS) dataset: In this dataset, sessions aimed to record from the same brain regions. This dataset has lower explicit brain region heterogeneity.
  • Brainwidemap (BWM) dataset: This dataset aimed to cover a wider variety of brain regions across sessions, leading to higher explicit brain region heterogeneity.

For RS, they used 89 sessions for pre-training and 5 for held-out evaluation. For BWM, they used 200 sessions for pre-training and 10 for held-out evaluation.

Key Insights and Ideas

  • Heterogeneity is a Bottleneck: Explicit heterogeneity, like recording from different brain regions, significantly hinders the scaling benefits of pre-training. The more varied the brain regions in the pre-training data, the less performance improvement you see on tasks like predicting neuron or region-level activity.
  • Not All Data is Equal: The study highlights that simply adding more data doesn't guarantee better performance. The quality and relevance of the pre-training data, particularly its similarity to the downstream task data, are crucial.
  • "Good" Sessions Matter More: A small number of carefully selected pre-training sessions can be far more effective than a large number of randomly chosen sessions. This suggests that implicit heterogeneity (subtle session-to-session differences) plays a significant role.
  • Challenging Traditional Scaling Laws: The findings question the direct applicability of "scaling laws" (where performance scales predictably with data size) to neuroscience data, suggesting that these laws might be too simplistic for complex, heterogeneous biological signals.

Key Results

  • Reduced Scaling Benefits: Greater brain region heterogeneity (in the BWM dataset compared to RS) led to reduced improvements in neuron- and region-level activity prediction performances. For example, scaling benefits for co-smoothing were negligible in BWM.
  • Effective Session Selection: Their proposed session-selection procedure, based on single-session finetuning performance, could identify highly beneficial sessions.
  • Outperforming Random Selection: Models trained on as few as five selected sessions outperformed models trained on randomly chosen sessions, even when the full dataset was used. This demonstrates a significant gain in data efficiency.
  • Forward-Prediction Scales Best: Among the tasks, forward-prediction showed the most consistent scaling behavior across both datasets, though it also plateaued around 80 pre-training sessions.
  • Choice Decoding Scaling is Weak: Decoding animal choices showed no clear scaling of performance with increased pre-training data on either dataset.

Limitations and Future Work

  • Limited Scope of Heterogeneity: While they explored explicit (brain region) and implicit (session-to-session) heterogeneity, other sources like animal-to-animal variability or different recording modalities were not as deeply investigated.
  • Focus on NDT: The study focused on the NDT architecture. Future work could explore if these findings hold for other architectures and learning objectives.
  • Computational Constraints: For the BWM dataset, they couldn't train larger models due to computational resource constraints, which might have influenced results.
  • Future Directions: The authors suggest future work should focus on:
  • Scaling experiments with different architectures and learning objectives (including supervised approaches).
  • Investigating other modalities of neural data (e.g., calcium imaging, local field potentials).
  • Developing more computationally efficient session selection strategies.
  • Advocating for rigorous scaling analyses in future neuroscience foundation model research to account for data heterogeneity.

Summary updated

Abstract

Abstract A key challenge in analyzing neuroscience datasets is the profound variability they exhibit across sessions, animals, and data modalities–i.e., heterogeneity. Several recent studies have demonstrated performance gains from pretraining neural foundation models on multi-session datasets, seemingly overcoming this challenge. However, these studies typically lack finegrained data scaling analyses. It remains unclear how different sources of heterogeneity influence model performance as the amount of pretraining data increases, and whether all sessions contribute equally to downstream performance gains. In this work, we systematically investigate how data heterogeneity impacts the scaling behavior of neural data transformers (NDTs) in neural activity prediction. We found that explicit sources of heterogeneity, such as brain region mismatches among sessions, reduced scaling benefits of neuron-level and region-level activity prediction performances. For tasks that do exhibit consistent scaling, we identified implicit data heterogeneity arising from cross-session variability. Through our proposed session-selection procedure, models pretrained on as few as five selected sessions outperformed those pretrained on the entire dataset of 84 sessions. Our findings challenge the direct applicability of traditional scaling laws to neural data and suggest that prior reports of multi-session scaling benefits may need to be re-examined in the light of data heterogeneity. This work both highlights the importance of incremental data scaling analyses and suggests new avenues toward optimally selecting pretraining data when developing foundation models on large-scale neuroscience datasets.
BibTeX
@inproceedings{jiang2025data,
  author = {Linxing Preston Jiang and Shirui Chen and Emmanuel Tanumihardja and Xiaochuang Han and Weijia Shi and Eric Shea‐Brown and Rajesh P. N. Rao},
  title = {{Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data Transformers}},
  booktitle = {LM4Sci @ COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=WiECjcxNw4},
}
Figure