XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging

Huy Phan, Oliver Y. Chén, Philipp Koch, Alfred Mertins, Maarten De VosIEEE TPAMI 2020 · DOI · OpenReview

TL;DR

This paper introduces XSleepNet, a novel deep learning model for automatic sleep staging that effectively combines two different types of input signals: raw polysomnography (PSG) data and time-frequency images derived from it. By adaptively adjusting the learning rate for each input "view" based on its generalization behavior, XSleepNet achieves superior performance compared to single-view models and simpler multi-view fusion methods, pushing the state-of-the-art in sleep staging.

What Problem Does This Paper Address?

Accurate sleep staging, the process of classifying sleep into different stages (e.g., wake, N1, N2, N3, REM), is crucial for diagnosing sleep disorders and understanding sleep's impact on health. Manual sleep staging by experts is time-consuming and labor-intensive, making automated methods essential for large-scale sleep assessment and longitudinal monitoring. While deep learning has shown promise, effectively leveraging multiple types of input signals (like raw physiological data and their time-frequency representations) for sleep staging has been challenging. Previous attempts to combine these "views" often resulted in performance degradation rather than improvement, and the reasons for this were not well understood. This paper aims to bridge this gap by proposing a principled approach to multi-view learning for sleep staging.

Methods

The core of XSleepNet is a sequence-to-sequence architecture designed to learn a joint representation from two input views: raw physiological signals and time-frequency images. The novelty lies in its adaptive gradient blending mechanism. Instead of simply concatenating features from different views, XSleepNet monitors the generalization and overfitting behavior of each view's classification branch during training. Based on these measures, it dynamically adjusts the contribution of each view's gradients to the overall learning process. This means the model learns faster from views that are generalizing well and slows down learning from views that are overfitting.

The architecture itself comprises two main streams:

  • Raw Signal Stream: A deep Convolutional Neural Network (CNN) designed to capture temporal patterns in the raw physiological data. This stream is intentionally made larger to handle the complexity of raw signals.
  • Time-Frequency Image Stream: A Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM) network, to process the time-frequency representations. This stream is designed to be more compact.

The paper also outlines design principles for XSleepNet:

  1. Robustness: Varying the model size of the two streams makes the overall system robust to different amounts of training data.
  2. Complementarity: Using different types of RNNs (GRU for raw, LSTM for time-frequency) and architectures aims to ensure the views capture distinct information.
  3. Generalization/Overfitting-Aware Training: This is the key adaptive gradient blending principle.

Two approximation approaches for this adaptive training are proposed: XSleepNet1 (first-order approximation) and XSleepNet2 (second-order approximation using tangent fitting), with XSleepNet2 aiming for more stable learning.

Datasets

The authors evaluated XSleepNet on five publicly available sleep databases:

  • SleepEDF-20: 20 subjects, R&K scoring.
  • SleepEDF-78: 78 subjects, R&K scoring.
  • MASS: 200 subjects, AASM/R&K scoring.
  • Physio2018: 994 subjects, AASM scoring.
  • SHHS: 5,791 subjects, R&K scoring.

These datasets vary in size, allowing the authors to demonstrate the robustness of their approach. The paper also mentions experiments using different channel combinations (EEG, EEG-EOG, EEG-EOG-EMG) and notes that standard data splits were used for evaluation.

Key Insights and Ideas

  • Multi-view learning is tricky: Simply combining features from different signal types (raw vs. time-frequency) doesn't automatically lead to better performance. In fact, it can sometimes hurt it.
  • Views learn at different rates: Different signal representations (raw signals, time-frequency images) can generalize or overfit at different speeds during training. This asynchronous learning is a key challenge.
  • Adaptive learning is the solution: By actively monitoring and adjusting the learning pace for each view based on its generalization/overfitting behavior, XSleepNet can effectively integrate information from multiple sources.
  • Architecture matters for complementarity: Designing streams with different architectures (CNN for raw, RNN for time-frequency) helps ensure they capture complementary information, rather than redundant features.
  • Robustness to data size: The proposed architecture, with its varying stream sizes, is designed to perform well regardless of whether the training dataset is small or large.

Key Results

  • State-of-the-art performance: XSleepNet consistently outperformed single-view baselines (ARNN+RNN, FCNN+RNN) and a naive fusion baseline across all five experimental databases and various channel combinations.
  • Significant accuracy gains: On average, XSleepNet1 improved overall accuracy by 0.9% to 1.1% over the baselines. XSleepNet2 achieved even higher gains, averaging 1.2% to 1.4%.
  • Improved robustness: XSleepNet demonstrated strong performance across datasets of varying sizes, addressing a key design principle.
  • Transfer learning success: A fine-tuned XSleepNet model also showed improvements over prior transfer learning approaches on a challenging dataset.
  • XSleepNet2's stability: The XSleepNet2 approach, using a second-order approximation for gradient blending, showed more stable learning behavior, especially on smaller datasets, compared to XSleepNet1.

Limitations and Future Work

  • Approximation stability: While XSleepNet2 is more stable than XSleepNet1, the paper notes that the adaptive loss weights can still be noisy. They suggest self-ensembling as a potential solution.
  • Model size comparison: Although XSleepNet has a smaller footprint than some prior models like DeepSleepNet, further exploration of optimal model sizes for different datasets could be beneficial.
  • Channel combination strategy: The paper raises an interesting question about whether to treat multiple EEG channels as separate views or combine them differently, suggesting this as a future research direction.
  • Generalizability: The authors believe the multi-view approach is generic and can be applied to other modalities and applications beyond sleep staging, such as audio/speech processing and computer vision, where multi-view data is common.

Summary updated

Abstract

Automating sleep staging is vital to scale up sleep assessment and diagnosis to serve millions experiencing sleep deprivation and disorders and enable longitudinal sleep monitoring in home environments. Learning from raw polysomnography signals and their derived time-frequency image representations has been prevalent. However, learning from multi-view inputs (e.g., both the raw signals and the time-frequency images) for sleep staging is difficult and not well understood. This work proposes a sequence-to-sequence sleep staging model, XSleepNet, that is capable of learning a joint representation from both raw signals and time-frequency images. Since different views may generalize or overfit at different rates, the proposed network is trained such that the learning pace on each view is adapted based on their generalization/overfitting behavior. In simple terms, the learning on a particular view is speeded up when it is generalizing well and slowed down when it is overfitting. View-specific generalization/overfitting measures are computed on-the-fly during the training course and used to derive weights to blend the gradients from different views. As a result, the network is able to retain the representation power of different views in the joint features which represent the underlying distribution better than those learned by each individual view alone. Furthermore, the XSleepNet architecture is principally designed to gain robustness to the amount of training data and to increase the complementarity between the input views. Experimental results on five databases of different sizes show that XSleepNet consistently outperforms the single-view baselines and the multi-view baseline with a simple fusion strategy. Finally, XSleepNet also outperforms prior sleep staging methods and improves previous state-of-the-art results on the experimental databases.
BibTeX
@article{phan2020xsleepnet,
  author = {Huy Phan and Oliver Y. Chén and Philipp Koch and Alfred Mertins and Maarten De Vos},
  title = {{XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2021.3070057},
}
Figure