Teacher Assistant-Based Knowledge Distillation Extracting Multi-level Features on Single Channel Sleep EEG

Heng Liang, Yucheng Liu, Haichao Wang, Ziyu JiaIJCAI 2023 · OpenReview

Abstract

Sleep stage classification is of great significance to the diagnosis of sleep disorders. However, existing sleep stage classification models based on deep learning are usually relatively large in size (wider and deeper), which makes them hard to be deployed on wearable devices. Therefore, it is a challenge to lighten the existing sleep stage classification models. In this paper, we propose a novel general knowledge distillation framework for sleep stage classification tasks called SleepKD. Our SleepKD, composed of the multi-level module, teacher assistant module, and other knowledge distillation modules, aims to lighten large-scale sleep stage classification models. Specifically, the multi-level module is able to transfer the multi-level knowledge extracted from sleep signals by the teacher model (large-scale model) to the student model (lightweight model). Moreover, the teacher assistant module bridges the large gap between the teacher and student network, and further improves the distillation. We evaluate our method on two public sleep datasets (Sleep-EDF and ISRUC-III). Compared to the baseline methods, the results show that our knowledge distillation framework achieves state-of-the-art performance. SleepKD can significantly lighten the sleep model while maintaining its classification performance.

Introduction

Automated sleep stage classification—categorizing sleep into stages like REM, N1, N2, N3, and Wake—is essential for diagnosing sleep disorders. While deep learning models have significantly improved accuracy, their complexity (wider and deeper architectures) poses a significant barrier to deployment on resource-constrained wearable devices like smartwatches or headbands.

Standard Knowledge Distillation (KD), where a compact "student" model mimics a large "teacher" model, often struggles in this domain due to two primary factors:

  1. Hierarchical Data Complexity: Sleep signals contain both local "epoch-level" features (e.g., sleep spindles) and global "sequence-level" features (transition patterns between stages).
  2. The Capacity Gap: A direct transfer from a very large teacher to a very small student often results in poor knowledge absorption because the architectural difference is too vast.

The SleepKD Framework

To address these challenges, the authors introduce SleepKD, a framework that leverages multi-level feature extraction and an intermediate "Teacher Assistant" to bridge the gap between complex and lightweight models.

Hierarchical feature extraction: Sleep signals are processed into epoch-level features, which are then aggregated into sequence-level features to capture transitions.

Core Components

The SleepKD architecture consists of three main modules designed to optimize the transfer of knowledge:

  1. Multi-Level Module: This module ensures the student model learns the hierarchical nature of sleep data. It aligns the student’s intermediate representations with the teacher’s at two levels:
  2. Epoch Learning: Capturing local characteristics within 30-second sleep segments.
  3. Sequence Learning: Capturing the transition rules between epochs.
  4. Teacher Assistant (TA) Module: To mitigate the capacity gap, a medium-sized TA model is introduced. The teacher first distills knowledge to the TA, which then distills it to the student, creating a smoother learning trajectory.
  5. Hybrid Loss Function: Training is guided by a combination of epoch-level distillation loss, sequence-level distillation loss, soft label distillation (KL divergence), and standard hard label cross-entropy.
The complete SleepKD framework architecture, illustrating the distillation paths from Teacher to Teacher Assistant to Student, including multi-level feature alignment.

Experimental Evaluation

The framework was tested on two public datasets: Sleep-EDF and ISRUC-III, using two distinct base architectures: SalientSleepNet (CNN-based) and DeepSleepNet (CNN-RNN hybrid).

Key Results

  • Performance Retention: On Sleep-EDF, the student model achieved 87.05% accuracy, nearly matching the teacher's performance while being significantly smaller.
  • Significant Compression: The framework achieved compression ratios of approximately 74.68% for SalientSleepNet and 71.78% for DeepSleepNet.
  • Efficiency: Student models demonstrated a 5.59x to 6.85x speedup in inference time compared to their respective teachers, making real-time processing on low-power hardware feasible.

Conclusion

SleepKD provides a robust solution for deploying high-performance sleep staging models on wearable hardware. By explicitly modeling the hierarchical structure of sleep data and using a Teacher Assistant to bridge the architectural gap, it achieves state-of-the-art compression without significant loss in classification accuracy. Future work may extend this multi-level distillation approach to other time-series tasks such as ECG or multi-modal sensor analysis.

Summary updated

Abstract

Sleep stage classification is of great significance to the diagnosis of sleep disorders. However, existing sleep stage classification models based on deep learning are usually relatively large in size (wider and deeper), which makes them hard to be deployed on wearable devices. Therefore, it is a challenge to lighten the existing sleep stage classification models. In this paper, we propose a novel general knowledge distillation framework for sleep stage classification tasks called SleepKD. Our SleepKD, composed of the multi-level module, teacher assistant module, and other knowledge distillation modules, aims to lighten large-scale sleep stage classification models. Specifically, the multi-level module is able to transfer the multi-level knowledge extracted from sleep signals by the teacher model (large-scale model) to the student model (lightweight model). Moreover, the teacher assistant module bridges the large gap between the teacher and student network, and further improves the distillation. We evaluate our method on two public sleep datasets (Sleep-EDF and ISRUC-III). Compared to the baseline methods, the results show that our knowledge distillation framework achieves state-of-the-art performance. SleepKD can significantly lighten the sleep model while maintaining its classification performance. The source code is available at https://github.com/HychaoWang/SleepKD.
BibTeX
@inproceedings{liang2023teacher,
  author = {Heng Liang and Yucheng Liu and Haichao Wang and Ziyu Jia},
  title = {{Teacher Assistant-Based Knowledge Distillation Extracting Multi-level Features on Single Channel Sleep EEG}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=CkuiNlpcdX},
}
Figure