Spatio-temporal transformers for decoding neural movement control

Benedetta Candelori, Giampiero Bardella, Indro Spinelli, Surabhi Ramawat, Pierpaolo Pani, Stefano Ferraina, Simone Scardapane

TL;DR

Researchers have developed a specialized Spatio-Temporal Transformer (ST-T) architecture capable of decoding single-neuron spiking activity to predict movement intentions with high precision. The model not only identifies movement direction within 230ms of a "Go" signal but can also forecast whether a subject will successfully inhibit a movement before a "Stop" signal is even presented.

What Problem Does This Paper Address?

High-resolution neurophysiological data—the kind recorded directly from the brain—is a gold mine for developing Brain-Machine Interfaces (BMIs). However, translating raw neural spikes into actionable commands for a prosthetic limb or computer cursor is notoriously difficult.

The field faces a "Goldilocks" problem: models need to be complex enough to capture the intricate dynamics of neural populations, yet efficient enough to work in "low-data regimes" (where we don't have millions of training examples). Furthermore, many deep learning models are "black boxes." In clinical or scientific settings, we need interpretability—we need to know why a model made a specific prediction to ensure it is relying on actual neural planning rather than noise.

Methods

The authors introduce the Spatio-Temporal Transformer (ST-T). While standard Transformers (like those powering ChatGPT) are designed for sequences of words, the ST-T is tailored for the "space-time" nature of neural data.

  • Tokenization: The model treats neural activity as a grid of "tokens." Depending on the layer, a token represents either a single neuron’s activity over time or the state of all neurons at a specific instant.
  • Alternating Attention: The architecture uses alternating blocks of Multi-Head Attention (MHA). One block focuses on temporal correlations (how a neuron's past activity influences its future), and the other focuses on spatial correlations (how different neurons interact with one another).
  • Causal Masking: For real-time forecasting, the authors used a "causal" variant of the model. This ensures the model only uses information from the past to predict the future, mimicking how a real-time BMI would function.

Datasets

The model was validated using multi-electrode recordings from the dorsal premotor cortex (PMd) of two rhesus macaque monkeys (Monkey P and Monkey C).

  • The Task: A "Stop-signal task." Monkeys were trained to reach for a target when a "Go" signal appeared but to refrain from moving if a subsequent "Stop" signal was presented.
  • The Data: Single-trial binary spike matrices (neurons x time) recorded via a 96-electrode array.

Key Insights and Ideas

  • Decoding the "Unseen": Perhaps the most striking insight is the model's ability to predict movement inhibition. It could accurately forecast a "Correct Stop" trial before the Stop signal actually appeared. This suggests the model is picking up on the internal "preparatory state" or level of caution in the monkey’s brain.
  • Attention as a Scientific Tool: In this paper, "attention maps" aren't just a machine learning mechanism; they serve as a proxy for neurophysiological synchronization. The authors found that as the data passes through deeper layers of the Transformer, the model builds increasingly complex maps of how neurons synchronize, mirroring established theoretical models of motor planning.
  • Robustness Across Subjects: One major hurdle in neuro-decoding is that every brain is different. While a standard Convolutional Neural Network (CNN) performed well on one monkey but poorly on the other, the ST-T remained consistently accurate across both subjects.

Key Results

  • Directional Accuracy: The ST-T achieved >90% accuracy in predicting movement direction (D1 vs. D2) in almost all sessions, significantly outperforming the CNN baseline.
  • Early Prediction: The model reached 80% decoding accuracy as early as 135ms to 204ms after the Go signal. For context, this is well before the physical movement actually begins.
  • Inhibition Prediction: In the difficult task of distinguishing between "Go" trials and "Correct Stop" trials, the ST-T achieved up to 95% accuracy, whereas the CNN struggled, particularly with Monkey C (dropping to ~71%).

Limitations and Future Work

While the results are promising, the study is limited to a specific brain region (PMd) and a relatively simple two-direction motor task. The authors acknowledge that more formal analysis is needed to fully bridge the gap between "attention maps" and biological "functional connectivity."

Future work involves applying this architecture to more complex, multi-dimensional movements and testing its performance in closed-loop BMI systems, where the model’s predictions directly control a robotic device in real-time. The authors also suggest that this "relational inference" approach could help scientists discover new types of interactions between neurons that traditional statistical methods might miss.

Summary updated

Abstract

<i>Objective</i>. Deep learning tools applied to high-resolution neurophysiological data have significantly progressed, offering enhanced decoding, real-time processing, and readability for practical applications. However, the design of artificial neural networks to analyze neural activity<i>in vivo</i>remains a challenge, requiring a delicate balance between efficiency in low-data regimes and the interpretability of the results.<i>Approach</i>. To address this challenge, we introduce a novel specialized transformer architecture to analyze single-neuron spiking activity. The model is tested on multi-electrode recordings from the dorsal premotor cortex of non-human primates performing a motor inhibition task.<i>Main results</i>. The proposed architecture provides an early prediction of the correct movement direction, achieving accurate results no later than 230 ms after the Go signal presentation across animals. Additionally, the model can forecast whether the movement will be generated or withheld before a stop signal, unattended, is actually presented. To further understand the internal dynamics of the model, we compute the predicted correlations between time steps and between neurons at successive layers of the architecture, with the evolution of these correlations mirrors findings from previous theoretical analyses.<i>Significance</i>. Overall, our framework provides a comprehensive use case for the practical implementation of deep learning tools in motor control research, highlighting both the predictive capabilities and interpretability of the proposed architecture.
BibTeX
@article{candelori2025spatio,
  author = {Benedetta Candelori and Giampiero Bardella and Indro Spinelli and Surabhi Ramawat and Pierpaolo Pani and Stefano Ferraina and Simone Scardapane},
  title = {{Spatio-temporal transformers for decoding neural movement control}},
  journal = {Journal of Neural Engineering},
  year = {2025},
  doi = {10.1088/1741-2552/adaef0},
}
Figure