TL;DR
This paper introduces a novel cross-modal retrieval system (CCHR) that links infrared (IR) video of sleeping individuals with their electroencephalogram (EEG) signals. By bridging the semantic gap between these modalities using contrastive learning and efficient hashing, CCHR enables accurate retrieval of corresponding EEG data from IR video queries, paving the way for more accessible home-based sleep monitoring.
What Problem Does This Paper Address?
Accurate sleep quality assessment and diagnosis of sleep disorders are crucial for overall health, but the current gold standard, Polysomnography (PSG), is cumbersome and impractical for widespread home use. While IR cameras offer a more convenient way to monitor sleep at home, extracting meaningful diagnostic information directly from video is challenging due to the inherent "semantic gap" between visual cues and the complex physiological signals captured by EEG. Furthermore, the large volume of sleep data necessitates efficient storage and retrieval methods. This paper aims to bridge this gap by developing a system that can effectively retrieve relevant EEG data based on IR video input, facilitating a more accessible and potentially automated approach to sleep monitoring and diagnosis.
Methods
The proposed Cross-modal Contrastive Hashing Retrieval (CCHR) system comprises two main modules:
- Cross-modal Feature Extraction Module: This module uses deep neural networks to extract deep semantic features from both IR video clips and EEG signals. Notably, the IR video encoder is based on a pre-trained 3D ResNet-18, while the EEG encoder utilizes an architecture inspired by AttnSleep. Crucially, these encoders are pre-trained on sleep stage classification and then fine-tuned for instance-level cross-modal retrieval.
- Contrastive Hashing Module: This module is designed to generate discriminative and compact binary hash codes for both modalities. It employs a novel contrastive learning strategy with "hard negative mining" to learn consistent cross-modal representations, effectively reducing the semantic gap. To address the storage and inference efficiency requirements, a contrastive hashing approach is used to transform these features into binary hash codes. The module is optimized using a combined loss function that includes:
- Cross-modal Contrastive Loss ($L_c$): This loss encourages similar cross-modal pairs to have close embeddings and dissimilar pairs to have distant embeddings, using a triplet loss formulation with hard negative samples.
- Quantization Loss ($L_q$): This loss minimizes the difference between the continuous binary-like codes and the discrete hash values.
- Bit Balance Loss ($L_{bb}$): This loss ensures that each bit in the hash code is utilized equally, promoting better representation.
Datasets
The primary dataset used in this research is the S³VE dataset, which was collected by the authors. This dataset is unique as it comprises synchronized infrared (IR) video and PSG (including EEG) data from 105 individuals. Existing sleep datasets typically only include EEG data. The S³VE dataset was collected from Peking University Third Hospital and includes individuals suspected of sleep disorders like Obstructive Sleep Apnea (OSA). For their experiments, the authors utilized 102 individuals and processed the data into clips, with a subset of 134,070 clips used for classification. For retrieval tasks, they randomly selected 10,000 clips as the query set and the remaining as the database.
Key Insights and Ideas
- Bridging the Semantic Gap: The core challenge is the significant semantic gap between visual IR data and physiological EEG signals. The paper's success hinges on effectively learning shared representations that capture the underlying sleep state, rather than just superficial visual or electrical patterns.
- Hard Negative Mining for Contrastive Learning: Using "hard" negative samples (those that are semantically similar but incorrectly paired) in contrastive learning is crucial for pushing the boundaries of representation learning and improving the discriminative power of the learned embeddings.
- Hashing for Efficiency: The sheer volume of sleep data makes traditional retrieval methods computationally expensive. Employing hashing transforms complex features into compact binary codes, enabling much faster search and storage, which is essential for practical applications.
- IR Video as a Complement to EEG: The research highlights that IR video, while not a direct diagnostic tool on its own, contains rich information (body movements, facial expressions, respiratory rhythms) that can be leveraged to infer sleep states and complement EEG data, especially in home-monitoring scenarios.
- Modality-Specific Strengths: The paper observes that IR video queries tend to yield slightly better retrieval results than EEG queries. This is attributed to the IR video's richer, more diverse information content, which can more directly correlate with specific sleep stage changes, whereas EEG signals might have subtle changes that are harder to visually interpret.
Key Results
The proposed CCHR method demonstrated superior performance compared to existing state-of-the-art cross-modal hashing retrieval methods on the S³VE dataset.
- MAP@1000 Performance: CCHR consistently outperformed baseline methods across different hash code lengths (16, 32, and 64 bits). For instance, with a 64-bit hash code, CCHR achieved a MAP@1000 of 0.592 when querying IR video against an EEG gallery, and 0.554 when querying EEG against an IR video gallery. This represents a significant improvement over methods like DUCH [51].
- Impact of Loss Components: An ablation study revealed that the quantization loss ($L_q$) had a more substantial impact on retrieval performance (approximately 3.6–6% improvement) compared to the bit balance loss ($L_{bb}$) (approximately 1.3–2.6% improvement). This suggests that accurately mapping continuous features to binary codes is more critical than ensuring bit balance for this specific task.
- Hyperparameter Sensitivity: The system showed reasonable sensitivity to hyperparameters, with optimal performance achieved for specific values of $\lambda_1$ (0.001) and $\lambda_2$ (0.01), and the contrastive learning margin $\alpha$ (0.25).
- Qualitative Analysis: Visualizations of retrieval results showed that CCHR could effectively retrieve similar sleep stages, with retrieved EEG signals matching the characteristics of the queried sleep stage (e.g., high-frequency waves for W stage, low-frequency for N3 stage).
Limitations and Future Work
- Data Collection Effort: The authors acknowledge that collecting synchronized sleep video and EEG data is labor-intensive.
- "Different Thresholds of Ease of Perception": The paper notes that IR video and EEG have different sensitivities to subtle changes, leading to a slight asymmetry in retrieval performance where IR video queries generally perform better. This difference, while minimized, cannot be entirely eliminated.
- Future Work:
- Telemedicine Assistance System: The authors aim to integrate CCHR into a telemedicine system for home-based sleep monitoring.
- Addressing Deep Sleep Similarity: Future work will focus on improving retrieval for deep sleep stages, where visual similarities can be high, potentially leading to similar retrieval results.
- Federated Learning: To overcome data privacy concerns and leverage diverse datasets, they plan to explore federated learning for cross-modal retrieval.
- Domain Adaptation: Improving the domain adaptation capability of existing algorithms is also a future direction.
Summary updated