Multi-block RNN Autoencoders Enable Broadband ECoG Signal Reconstruction

Michael Nolan, Bijan Pesaran, Eli Shlizerman, Amy L. OrsbornbioRxiv 2022 · DOI

TL;DR

This paper introduces two novel Recurrent Neural Network (RNN) autoencoder architectures, RAE and MRAE, designed to reconstruct broadband electrocorticography (ECoG) signals directly from raw time-series data. The proposed Multi-block RAE (MRAE) significantly improves upon the standard RAE by overcoming time-bandwidth limitations, enabling more accurate and broadband reconstructions of neural activity, which is crucial for advanced brain-computer interfaces and neuroscience research.

What Problem Does This Paper Address?

Reconstructing neural signals, particularly complex, high-dimensional time-series data like electrocorticography (ECoG), is a fundamental challenge in neuroscience and neural engineering. While previous methods using RNN autoencoders have shown success in reconstructing neuronal spiking data or specific features of neural signals (like power in certain frequency bands), they have struggled to directly reconstruct the full, broadband ECoG signal while preserving its phase and temporal resolution. This limitation hinders the development of sophisticated tools for understanding neural dynamics and building advanced brain-computer interfaces that rely on detailed, real-time neural signal information. The core problem is the inherent trade-off between the temporal duration of the signal that can be accurately reconstructed and the bandwidth (frequency range) of that reconstruction.

Methods

The authors propose two encoder-decoder RNN architectures:

  • RNN Autoencoder (RAE): This is a standard bidirectional GRU-based encoder-decoder model. It takes the entire input ECoG trial, encodes it into a latent representation, and then decodes it to reconstruct the original signal. The optimization objective is to minimize the Mean Squared Error (MSE) between the reconstructed and target signals. This model is a direct adaptation of existing methods for spiking data, but applied to continuous ECoG signals.
  • Multi-block RAE (MRAE): This is the novel contribution. MRAE addresses the limitations of RAE by decomposing the input ECoG signal into multiple non-overlapping frequency bands. Each frequency band is then processed by a separate RAE "block" (an autoencoder). The outputs from these individual blocks are then combined through a final fully-connected layer to produce the broadband reconstruction. This parallel processing allows each block to specialize in a specific frequency range, overcoming the time-bandwidth limitations of a single block. The optimization involves minimizing both individual block reconstruction errors and the overall broadband reconstruction error.

Datasets

The models were trained and evaluated on multi-channel micro-ECoG (μECoG) recordings from the motor cortices of a rhesus macaque. The data was collected over four days during unconstrained behavior. The original recordings were at 30kHz on 62 channels, which were then downsampled to 250Hz and filtered. After preprocessing and data rejection, the dataset comprised μECoG signal data from 41 channels, totaling approximately 33 hours of data. This dataset is not new but is utilized for training and evaluating the proposed models.

Key Insights and Ideas

  • Time-Bandwidth Trade-off: A single RNN autoencoder block has a fixed capacity for reconstructing signals, leading to a trade-off: longer sequences can be reconstructed with lower accuracy or a narrower frequency bandwidth.
  • Spectral Decomposition is Key: Breaking down the broadband signal into frequency bands and processing them in parallel (MRAE) effectively bypasses the time-bandwidth limitation of single-block models.
  • MRAE's Superiority for Broadband: MRAE demonstrates significantly better reconstruction accuracy and bandwidth, especially for longer ECoG sequences, compared to the standard RAE.
  • Importance of Phase and Temporal Resolution: The paper emphasizes the need for direct time-series reconstruction that preserves signal phase and temporal resolution, which is crucial for understanding neural dynamics and enabling advanced applications.
  • Model Size vs. Architecture: While increasing the number of units in a single RAE model can improve accuracy, it doesn't solve the fundamental bandwidth limitations. MRAE achieves better bandwidth with a more scalable architecture.

Key Results

  • RAE: Could reconstruct ECoG signals but was band-limited, performing better on lower frequencies and shorter time segments. Reconstruction accuracy decreased significantly with increasing trial length.
  • MRAE: Overcame the band-limiting issue of RAE, yielding broadband (0-100 Hz) and accurate reconstructions. MRAE maintained higher reconstruction bandwidth across longer trial lengths (up to 1000ms) compared to RAE.
  • Quantitative Performance:
    • For trial lengths of 400ms or greater, MRAE showed lower Mean Squared Error (MSE) than RAE.
  • Reconstruction bandwidth for MRAE was consistently higher than RAE, especially for longer sequences. For example, at 1000ms, MRAE achieved a bandwidth of ~76Hz, while RAE (512 unit) was around ~52Hz.
  • Even when comparing a larger RAE model (1536 units) to the MRAE (3 blocks, 512 units each), MRAE still outperformed in terms of reconstruction bandwidth, despite the larger RAE sometimes showing comparable or slightly better MSE.

Limitations and Future Work

  • Band-limited Reconstruction within Blocks: Even within MRAE, individual blocks are still biased towards reconstructing lower frequencies within their assigned band more accurately than higher frequencies.
  • Potential Causes for Limitations: The authors hypothesize that limitations might stem from the inductive bias of GRU RNNs, the power spectral density characteristics of ECoG (1/f distribution), and the L2 norm optimization objective, which prioritizes larger amplitude deviations in low frequencies.
  • Future Work:
    • Exploring alternative training objective functions (e.g., L1 norm, log-scale frequency domain error) to mitigate the bias towards low frequencies.
  • Investigating further extensions of the multiblock architecture for other modalities (e.g., MUA, LFP, EEG) and for cross-modal analysis.
  • Developing methods to capture cross-band interactions, such as cross-frequency coupling.
  • Applying these models for applications like data completion, forecasting, and anomaly detection in neural data.
  • Further exploring the source of time-bandwidth capacities for more general autoencoder modeling.

Summary updated

Abstract

Abstract Objective Neural dynamical models reconstruct neural data using dynamical systems. These models enable direct reconstruction and estimation of neural time-series data as well as estimation of neural latent states. Nonlinear neural dynamical models using recurrent neural networks in an encoder-decoder architecture have recently enabled accurate single-trial reconstructions of neural activity for neuronal spiking data. While these models have been applied to neural field potential data, they have only so far been applied to signal feature reconstruction (e.g. frequency band power), and have not yet produced direct reconstructions of broadband time-series data preserving signal phase and temporal resolution. Approach Here we present two encoder-decoder model architectures - the RNN autoencoder (RAE) and multi-block RAE (MRAE) for direct time-series reconstruction of broadband neural data. We trained and tested models on multi-channel micro-Electricorticography ( μ ECoG) recordings from non-human primate motor corticies during unconstrained behavior. Main Results We show that RAE reconstructs micro-electrocorticography recordings, but has reconstruction accuracy that is band-limited to model scale. The MRAE architecture overcomes these time-bandwidth restrictions, yielding broadband (0-100 Hz), accurate reconstructions of μ ECoG data. Significance RAE and MRAE reconstruct broadband μ ECoG data through multiblock dynamical modeling. The MRAE overcomes time-bandwitdh restrictions to provide improved accuracy for long time duration signals. The reconstruction capabilities provided by these models for broadband neural signals like μ ECoG may enable the development of improved tools and analysis for basic scientific research and applications like brain-computer interfaces.
BibTeX
@misc{nolan2022multi,
  author = {Michael Nolan and Bijan Pesaran and Eli Shlizerman and Amy L. Orsborn},
  title = {{Multi-block RNN Autoencoders Enable Broadband ECoG Signal Reconstruction}},
  year = {2022},
  doi = {10.1101/2022.09.07.507004},
  howpublished = {bioRxiv},
}
Figure