Spatiotemporally Resolved Multivariate Pattern Analysis for M/EEG

Cameron Higgins, Diego Vidaurre, Nils Kolling, Yunzhe Liu, Timothy E.j. Behrens, Mark W. WoolrichbioRxiv 2021 · DOI

TL;DR

This paper introduces SpatioTemporally Resolved Multivariate Pattern Analysis (STRM), a novel framework for analyzing M/EEG data that explicitly models both the spatial distribution and the precise timing of neural activity patterns on a trial-by-trial basis. STRM offers more interpretable insights into brain dynamics and achieves competitive or superior decoding accuracy compared to existing methods, paving the way for more flexible experimental designs.

What Problem Does This Paper Address?

Neuroscientists are increasingly interested in understanding how information is represented and processed in the brain over time. While M/EEG (Magnetoencephalography/Electroencephalography) offers excellent millisecond-level temporal resolution, traditional decoding methods often obscure the underlying spatial and temporal patterns, making them difficult to interpret. These methods typically assume that neural processes occur at the exact same millisecond on every trial, which limits the types of experimental designs that can be used and prevents the investigation of meaningful temporal variations across trials. This paper aims to bridge this gap by providing a method that can simultaneously infer spatial patterns and their precise timing, offering a more holistic understanding of brain dynamics.

Methods

The core of this paper is the SpatioTemporally Resolved MVPA (STRM) framework. STRM builds upon the General Linear Model (GLM) by incorporating two key innovations:

  1. Latent Markov Variable: A Hidden Markov Model (HMM) is used to model the temporal dynamics, allowing for the inference of discrete, sequential "states" of neural activity that emerge and evolve over time within each trial. This explicitly models the timing of these states.
  2. Multivariate Spatial Modeling: The model simultaneously infers the multivariate spatial distribution of activity for each identified state across all recorded channels.

This approach allows for the fitting of a generative encoding model, which can then be inverted using Bayes' rule to perform decoding (i.e., predict stimuli from brain activity). The authors also propose a sequential latent state prior, constraining state transitions to a fixed order (e.g., state 1 to state 2, then state 2 to state 3, etc.), which helps to regularize the model and prevent overfitting, especially in supervised learning scenarios.

Datasets

The authors evaluated their STRM framework on two distinct datasets:

  1. MEG Visual Data: This dataset involved participants viewing randomly presented visual images. It was used to test the STRM-Classification model.
  2. EEG Data: This dataset was collected during a decision-making and reward learning task where participants navigated foraging patches. It was used to test the STRM-Regression model, as the task involved continuous variables (e.g., reward value).

Key Insights and Ideas

  • Beyond Millisecond-Perfect Alignment: The assumption that neural processes must align perfectly in time across trials is a significant limitation. STRM relaxes this, allowing for the investigation of how timing varies and what influences it.
  • Interpretable Spatial Patterns: By fitting an encoding model first, STRM provides interpretable maps of how neural representations are spatially distributed for each identified state, offering a clearer link between brain activity and behavior.
  • Behaviorally Relevant Timing: The timing of these neural states is not random; it systematically varies with behavioral and physiological variables, suggesting that these timing differences reflect meaningful cognitive processes.
  • Generative Models for Interpretability: The paper advocates for generative encoding models as a superior approach for interpretability compared to inverting decoding models, especially when regularization is involved.

Key Results

  • Competitive Decoding Accuracy: STRM achieved classification accuracies that were equivalent to or modestly better than conventional methods, including timepoint-by-timepoint LDA.
  • Identification of Visual Processing Stages: In the MEG data, STRM successfully identified sequential stages of visual processing, with distinct spatial patterns emerging at different time points after stimulus onset, progressing from occipital to temporal areas.
  • Modulation of Timing by Behavior: In the EEG reward learning task, the timing of value processing states was significantly modulated by cognitive variables like accrual time and recent reward history, demonstrating the behavioral relevance of the inferred timings.
  • Robustness to Parameter Choice: The performance of STRM was robust across a range of parameters, particularly the number of states (K), suggesting flexibility in model selection.

Limitations and Future Work

  • Trade-offs in Predictive Accuracy: While STRM offers interpretability, some optimized sliding window techniques achieved comparable or slightly better predictive accuracy. This might be due to the estimation procedure for state timecourses in held-out test data, which can introduce error.
  • Assumption of Mutual Exclusivity: The model assumes discrete, mutually exclusive states, which can lead to sharp jumps in inferred activity patterns. If underlying neural processes are smoother, this discretization might introduce errors at state boundaries.
  • Generalizability of Findings: The authors suggest that the benefits of generative models for interpretability and accuracy, as demonstrated in this study, may generalize well to other M/EEG datasets.
  • Flexible Experimental Designs: The ability to model trial-to-trial variability in timing opens doors for more complex and flexible experimental designs, particularly for tasks involving higher cognitive functions with naturally variable onset timings.

Summary updated

Abstract

ABSTRACT An emerging goal in neuroscience is tracking what information is represented in brain activity over time as a participant completes some task. Whilst EEG and MEG offer millisecond temporal resolution of how activity patterns emerge and evolve, standard decoding methods present significant barriers to interpretability as they obscure the underlying spatial and temporal activity patterns. We instead propose the use of a generative encoding model framework that simultaneously infers the multivariate spatial patterns of activity and the variable timing at which these patterns emerge on individual trials. An encoding model inversion allows predictions to be made about unseen test data in the same way as in standard decoding methodology. These SpatioTemporally Resolved MVPA (STRM) models can be flexibly applied to a wide variety of experimental paradigms, including classification and regression tasks. We show that these models provide insightful maps of the activity driving predictive accuracy metrics; demonstrate behaviourally meaningful variation in the timing of pattern emergence on individual trials; and achieve predictive accuracies that are either equivalent or surpass those achieved by more widely used methods. This provides a new avenue for investigating the brain’s representational dynamics and could ultimately support more flexible experimental designs in future. HIGHLIGHTS We introduce SpatioTemporally Resolved MVPA (STRM), an approach that explicitly models how successive stages of stimulus processing are distributed in both space and time in M/EEG data. We show that STRM is broadly applicable to diverse types of M/EEG data and outputs meaningful and interpretable maps of how neural representations evolve in space and time at millisecond resolution. The trial-specific deviations in activity pattern timings identified by STRM are not random, but vary systematically with inter-trial differences in behavioural, cognitive and physiological variables. These methods result in predictive accuracy metrics that are mostly equivalent to, or a modest improvement on, conventional methods.
BibTeX
@misc{higgins2021spatiotemporally,
  author = {Cameron Higgins and Diego Vidaurre and Nils Kolling and Yunzhe Liu and Timothy E.J. Behrens and Mark W. Woolrich},
  title = {{Spatiotemporally Resolved Multivariate Pattern Analysis for M/EEG}},
  year = {2021},
  doi = {10.1101/2021.08.17.456594},
  howpublished = {bioRxiv},
}

Loading related content...

Figure