NLP4Neuro: Sequence-to-sequence learning for neural population decoding

Jacob J. Morra, Kaitlyn E. Fouke, Kexin Hang, Zichen He, Owen Traubert, Timothy W. Dunn, Eva A. NaumannarXiv 2025 · DOI · arXiv

Abstract

A fundamental challenge in neuroscience is decoding complex, high-dimensional neural activity into behavioral outputs. We introduce NLP4Neuro, a framework that repurposes Large Language Models (LLMs) to predict zebrafish movements from brain-wide calcium imaging. Our findings reveal that LLMs pre-trained on massive text corpora significantly outperform traditional recurrent neural networks and untrained transformers. Specifically, Mixture-of-Experts (MoE) architectures like DeepSeek-c7b demonstrate superior performance in capturing long-range temporal dependencies. These results suggest that the sequence-to-sequence patterns inherent in natural language share structural commonalities with neural population dynamics, enabling effective transfer learning for neural decoding.

Introduction

The goal of neural decoding is to translate the firing patterns of thousands of neurons into an animal's actual behavior. This task is complicated by the high dimensionality and noise inherent in neural data, which unfolds across multiple timescales. While Recurrent Neural Networks (RNNs) have been the standard for this task, they often struggle to scale or capture long-range dependencies effectively. Inspired by the success of Transformers in natural language processing, we investigate whether these models can be adapted to "read" the language of the brain.

Methods

The NLP4Neuro pipeline consists of two primary stages: model building and interpretation. We evaluated several Transformer architectures, including BERT (encoder-only), GPT-2 (decoder-only), and DeepSeek-c7b (a Mixture-of-Experts model). These models were tasked with predicting tail kinematics from volumetric two-photon calcium imaging of approximately 15,000 neurons in larval zebrafish, sampled at 1.1 Hz.

Experimental setup and NLP4Neuro pipeline overview.

The input neural data was packaged using various embedding strategies, including linear mappings and positional encodings. To ground our results, we utilized the ZAPBench benchmark for zebrafish activity prediction. To interpret the models, we employed salience mapping—calculating the gradient of the output with respect to the input—to identify the specific neurons most critical for behavioral prediction.

Results

A key discovery was the efficacy of transfer learning. LLMs initialized with pre-trained text weights consistently achieved lower Root-Mean-Squared Error (RMSE) than those trained from scratch. This suggests that the statistical structures learned from human language—such as syntax and long-range dependencies—generalize to the dynamics of neural populations.

Performance comparison between untrained and pre-trained LLMs.

The DeepSeek-c7b model emerged as the top performer, particularly at longer sequence lengths ($s=20$ frames). Qualitatively, LLMs were significantly better at capturing the precise timing of tail "bursts" compared to traditional RNNs. Furthermore, salience mapping identified neurons in the Pretectum (Pt) as highly influential, aligning with known biological roles of the Pt in visual motion processing and motor initiation in zebrafish.

Discussion and Future Work

NLP4Neuro demonstrates that off-the-shelf LLMs can be powerful tools for neuroscience. However, several limitations remain. The current study utilized "evoked" behavior driven by visual stimuli; future research must determine if these models can decode spontaneous, internally generated activity. Additionally, while salience maps provide correlational insights, causal validation through techniques like optogenetics is necessary to confirm the functional roles of identified neurons. Despite these challenges, the ability to leverage pre-trained AI models opens new avenues for understanding the complex relationship between brain activity and behavior.

Summary updated

BibTeX
@misc{morra2025nlp,
  author = {Jacob J. Morra and Kaitlyn E. Fouke and Kexin Hang and Zichen He and Owen Traubert and Timothy W. Dunn and Eva A. Naumann},
  title = {{NLP4Neuro: Sequence-to-sequence learning for neural population decoding}},
  year = {2025},
  eprint = {2507.02264},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.02264},
}
Figure