The ‘Sandwich’ meta-framework for architecture agnostic deep privacy-preserving transfer learning for non-invasive brainwave decoding

Xiaoxi Wei, Jyotindra Narayan, A. Aldo FaisalJournal of Neural Engineering 2024 · DOI

TL;DR

This paper introduces the "Sandwich" meta-framework, a novel approach that combines transfer learning and federated learning to address challenges in EEG brainwave decoding. It achieves superior performance on heterogeneous datasets while preserving user privacy, paving the way for more robust and scalable brain-computer interfaces.

What Problem Does This Paper Address?

Decoding human behavior from non-invasive brainwave signals like EEG is crucial for applications ranging from aiding paralyzed patients to understanding neural activity. However, two major hurdles exist:

  1. Data Variability: EEG data is notoriously inconsistent due to differences in sensor placement, individual brain anatomy, and experimental setups across datasets. This variability makes it difficult to train models that generalize well.
  2. Privacy Concerns: As we move towards larger-scale EEG analysis using data from multiple individuals and centers, protecting sensitive brainwave data becomes paramount. Sharing raw data across different entities is restricted by regulations and ethical considerations.

Existing methods often struggle to effectively handle both data variability and privacy simultaneously, limiting the potential of large-scale EEG analysis.

Methods

The core contribution is the "Sandwich" meta-framework, a three-layer architecture designed for privacy-preserving transfer learning:

  • Federated Networks (First Layers): These are individual "branches" or feature extractors for each data set. They are trained locally and handle initial data discrepancies (e.g., sensor differences, impedance). Crucially, they act as an "auto-encryption" mechanism, sharing only the extracted features with the central server, thus preserving raw data and parameter-level privacy.
  • Shared Network (Middle Layer): This layer learns common rules and patterns across all data sets. It receives the "encrypted" features from the local branches and applies transfer learning techniques. The paper explores two such techniques:
  • Maximum Mean Discrepancy (MMD): Aims to align the distributions of features from different data sets.
  • Deep Set: A technique that processes sets of data to enhance feature invariance and robustness, particularly useful for handling subject-specific variations.
  • Individual Classifiers (Final Layers): These layers are responsible for the final decoding task. The framework supports two configurations: a single unified classifier for all tasks or multiple individual classifiers, offering flexibility for tasks with different label sets.

The framework is designed to be model-agnostic, meaning it can integrate various existing EEG decoding models (like Shallow ConvNet and EEGInception) as its feature extractors.

Datasets

The research utilizes the BEETL motor imagery (MI) challenge dataset, which is a benchmark for heterogeneous EEG data. This dataset comprises:

  • Source Datasets:
    • Cho2017: 52 subjects, left/right hand MI.
  • BCICIV2a: 9 subjects, left/right hand, feet, tongue MI.
  • PhysioMI: 109 subjects, left/right hand, feet, both hands, rest MI.
  • Weibo2014: 10 subjects, left/right hand, feet, rest MI.
  • CybathlonIC: 5 subjects, left/right hand, feet, rest MI.
  • Test Set: Comprised of 5 subjects from two different datasets (CybathlonIC and Weibo2014), with the final 200 trials of each subject used for evaluation.

The paper notes that for training, datasets were balanced to 2880 trials, with PhysioMI and target data augmented by duplication to match this size.

Key Insights and Ideas

  • Privacy as a First-Class Citizen: The "Sandwich" architecture inherently builds privacy into the transfer learning process, moving beyond just securing the final model. Local feature extraction and sharing only processed features are key to this.
  • Model-Agnostic Meta-Framework: The ability to plug in different backbone architectures (like Shallow ConvNet and EEGInception) and transfer learning techniques (MMD, Deep Set) makes the "Sandwich" a versatile tool for various time-series decoding tasks.
  • Balancing Alignment and Specificity: The paper highlights a trade-off: MMD aims for broad distribution alignment, which can be beneficial for similar tasks but might dilute unique dataset features. Deep Set, on the other hand, enhances subject-specific invariance, which is more robust for complex, heterogeneous data.
  • Interpretability through Network Focus: Visualizations (like t-SNE and network focus maps) help understand how the model learns and aligns features, showing that the framework can both generalize across datasets and retain discriminative information for specific tasks.

Key Results

  • Superior Performance: The "Sandwich" framework consistently outperformed baseline models.
  • The best-performing model, Inception-SD-Deepset-MultiCls, achieved a 9% improvement over the EEGInception baseline.
  • ShallowConv-SD-Deepset-MultiCls surpassed its baseline by 6%.
  • Privacy Preservation: The framework successfully preserved data-level, parameter-level, and inference-level privacy by keeping raw data and individual classifiers local.
  • Flexibility: The "DiffCh" experiment demonstrated the framework's ability to handle varying input channel sizes, achieving 68% accuracy, highlighting its adaptability.

Limitations and Future Work

  • Data Imbalance: The paper acknowledges the challenge of disparate sample sizes across datasets and how they addressed it through balancing and augmentation. Future work could explore more sophisticated methods for handling imbalanced sets.
  • Synchronous vs. Asynchronous Training: The current study focuses on theoretical applicability. Real-world implementation would require addressing asynchronous communication and training in federated settings.
  • Generalization to Other Data Types: While promising for EEG, the framework's applicability to other time-series data or modalities like fMRI (which is more image-like) needs further investigation and potential refinement.
  • Overfitting Noise: The study noted that models might learn noisy features. Future research could explore methods for removing such noisy features to enhance generalizability.
  • Choice of Transfer Learning Method: The paper suggests that the choice between MMD and Deep Set depends on the degree of distributional divergence between datasets, indicating a need for further research into optimal selection criteria.

Summary updated

Abstract

<i>Objective</i>. Machine learning has enhanced the performance of decoding signals indicating human behaviour. Electroencephalography (EEG) brainwave decoding, as an exemplar indicating neural activity and human thoughts non-invasively, has been helpful in neural activity analysis and aiding paralysed patients via brain-computer interfaces. However, training machine learning algorithms on EEG encounters two primary challenges: variability across data sets and privacy concerns using data from individuals and data centres. Our objective is to address these challenges by integrating transfer learning for data variability and federated learning for data privacy into a unified approach.<i>Approach</i>. We introduce the 'Sandwich' as a novel deep privacy-preserving meta-framework combining transfer learning and federated learning. The 'Sandwich' framework comprises three components: federated networks (first layers) that handle data set differences at the input level, a shared network (middle layer) learning common rules and applying transfer learning techniques, and individual classifiers (final layers) for specific brain tasks of each data set. This structure enables the central network (central server) to benefit from multiple data sets, while local branches (local servers) maintain data and label privacy.<i>Main results</i>. We evaluated the 'Sandwich' meta-architecture in various configurations using the BEETL motor imagery challenge, a benchmark for heterogeneous EEG data sets. Compared with baseline models like<i>Shallow ConvNet</i>and<i>EEGInception</i>, our 'Sandwich' implementations showed superior performance. The best-performing model, the Inception SanDwich with deep set alignment (<i>Inception-SD-Deepset</i>), exceeded baseline methods by 9%.<i>Significance</i>. The 'Sandwich' framework demonstrates advancements in federated deep transfer learning for diverse tasks and data sets. It outperforms conventional deep learning methods, showcasing the potential for effective use of larger, heterogeneous data sets with enhanced privacy. In addition, through its diverse implementations with various backbone architectures and transfer learning approaches, the 'Sandwich' framework shows the potential as a model-agnostic meta-framework for decoding time series data like EEG, suggesting a direction towards large-scale brainwave decoding by combining deep transfer learning with privacy-preserving federated learning.
BibTeX
@article{wei2024sandwich,
  author = {Xiaoxi Wei and Jyotindra Narayan and A. Aldo Faisal},
  title = {{The ‘Sandwich’ meta-framework for architecture agnostic deep privacy-preserving transfer learning for non-invasive brainwave decoding}},
  journal = {Journal of Neural Engineering},
  year = {2024},
  doi = {10.1088/1741-2552/ad9957},
}
Figure