Cross-Modal Contrastive Hashing Retrieval for Infrared Video and EEG

Jianan Han, Shaoxing Zhang, Aidong Men, Qingchao ChenSensors 2022 · OpenReview

TL;DR

This paper introduces a novel cross-modal retrieval system (CCHR) that links infrared (IR) video of sleeping individuals with their electroencephalogram (EEG) signals. By bridging the semantic gap between these modalities using contrastive learning and efficient hashing, CCHR enables accurate retrieval of corresponding EEG data from IR video queries, paving the way for more accessible home-based sleep monitoring.

What Problem Does This Paper Address?

Accurate sleep quality assessment and diagnosis of sleep disorders are crucial for overall health, but the current gold standard, Polysomnography (PSG), is cumbersome and impractical for widespread home use. While IR cameras offer a more convenient way to monitor sleep at home, extracting meaningful diagnostic information directly from video is challenging due to the inherent "semantic gap" between visual cues and the complex physiological signals captured by EEG. Furthermore, the large volume of sleep data necessitates efficient storage and retrieval methods. This paper aims to bridge this gap by developing a system that can effectively retrieve relevant EEG data based on IR video input, facilitating a more accessible and potentially automated approach to sleep monitoring and diagnosis.

Methods

The proposed Cross-modal Contrastive Hashing Retrieval (CCHR) system comprises two main modules:

  • Cross-modal Feature Extraction Module: This module uses deep neural networks to extract deep semantic features from both IR video clips and EEG signals. Notably, the IR video encoder is based on a pre-trained 3D ResNet-18, while the EEG encoder utilizes an architecture inspired by AttnSleep. Crucially, these encoders are pre-trained on sleep stage classification and then fine-tuned for instance-level cross-modal retrieval.
  • Contrastive Hashing Module: This module is designed to generate discriminative and compact binary hash codes for both modalities. It employs a novel contrastive learning strategy with "hard negative mining" to learn consistent cross-modal representations, effectively reducing the semantic gap. To address the storage and inference efficiency requirements, a contrastive hashing approach is used to transform these features into binary hash codes. The module is optimized using a combined loss function that includes:
  • Cross-modal Contrastive Loss ($L_c$): This loss encourages similar cross-modal pairs to have close embeddings and dissimilar pairs to have distant embeddings, using a triplet loss formulation with hard negative samples.
  • Quantization Loss ($L_q$): This loss minimizes the difference between the continuous binary-like codes and the discrete hash values.
  • Bit Balance Loss ($L_{bb}$): This loss ensures that each bit in the hash code is utilized equally, promoting better representation.

Datasets

The primary dataset used in this research is the S³VE dataset, which was collected by the authors. This dataset is unique as it comprises synchronized infrared (IR) video and PSG (including EEG) data from 105 individuals. Existing sleep datasets typically only include EEG data. The S³VE dataset was collected from Peking University Third Hospital and includes individuals suspected of sleep disorders like Obstructive Sleep Apnea (OSA). For their experiments, the authors utilized 102 individuals and processed the data into clips, with a subset of 134,070 clips used for classification. For retrieval tasks, they randomly selected 10,000 clips as the query set and the remaining as the database.

Key Insights and Ideas

  • Bridging the Semantic Gap: The core challenge is the significant semantic gap between visual IR data and physiological EEG signals. The paper's success hinges on effectively learning shared representations that capture the underlying sleep state, rather than just superficial visual or electrical patterns.
  • Hard Negative Mining for Contrastive Learning: Using "hard" negative samples (those that are semantically similar but incorrectly paired) in contrastive learning is crucial for pushing the boundaries of representation learning and improving the discriminative power of the learned embeddings.
  • Hashing for Efficiency: The sheer volume of sleep data makes traditional retrieval methods computationally expensive. Employing hashing transforms complex features into compact binary codes, enabling much faster search and storage, which is essential for practical applications.
  • IR Video as a Complement to EEG: The research highlights that IR video, while not a direct diagnostic tool on its own, contains rich information (body movements, facial expressions, respiratory rhythms) that can be leveraged to infer sleep states and complement EEG data, especially in home-monitoring scenarios.
  • Modality-Specific Strengths: The paper observes that IR video queries tend to yield slightly better retrieval results than EEG queries. This is attributed to the IR video's richer, more diverse information content, which can more directly correlate with specific sleep stage changes, whereas EEG signals might have subtle changes that are harder to visually interpret.

Key Results

The proposed CCHR method demonstrated superior performance compared to existing state-of-the-art cross-modal hashing retrieval methods on the S³VE dataset.

  • MAP@1000 Performance: CCHR consistently outperformed baseline methods across different hash code lengths (16, 32, and 64 bits). For instance, with a 64-bit hash code, CCHR achieved a MAP@1000 of 0.592 when querying IR video against an EEG gallery, and 0.554 when querying EEG against an IR video gallery. This represents a significant improvement over methods like DUCH [51].
  • Impact of Loss Components: An ablation study revealed that the quantization loss ($L_q$) had a more substantial impact on retrieval performance (approximately 3.6–6% improvement) compared to the bit balance loss ($L_{bb}$) (approximately 1.3–2.6% improvement). This suggests that accurately mapping continuous features to binary codes is more critical than ensuring bit balance for this specific task.
  • Hyperparameter Sensitivity: The system showed reasonable sensitivity to hyperparameters, with optimal performance achieved for specific values of $\lambda_1$ (0.001) and $\lambda_2$ (0.01), and the contrastive learning margin $\alpha$ (0.25).
  • Qualitative Analysis: Visualizations of retrieval results showed that CCHR could effectively retrieve similar sleep stages, with retrieved EEG signals matching the characteristics of the queried sleep stage (e.g., high-frequency waves for W stage, low-frequency for N3 stage).

Limitations and Future Work

  • Data Collection Effort: The authors acknowledge that collecting synchronized sleep video and EEG data is labor-intensive.
  • "Different Thresholds of Ease of Perception": The paper notes that IR video and EEG have different sensitivities to subtle changes, leading to a slight asymmetry in retrieval performance where IR video queries generally perform better. This difference, while minimized, cannot be entirely eliminated.
  • Future Work:
    • Telemedicine Assistance System: The authors aim to integrate CCHR into a telemedicine system for home-based sleep monitoring.
  • Addressing Deep Sleep Similarity: Future work will focus on improving retrieval for deep sleep stages, where visual similarities can be high, potentially leading to similar retrieval results.
  • Federated Learning: To overcome data privacy concerns and leverage diverse datasets, they plan to explore federated learning for cross-modal retrieval.
  • Domain Adaptation: Improving the domain adaptation capability of existing algorithms is also a future direction.

Summary updated

Abstract

It is essential to estimate the sleep quality and diagnose the clinical stages in time and at home, because they are closely related to and important causes of chronic diseases and daily life dysfunctions. However, the existing “gold-standard” sensing machine for diagnosis (Polysomnography (PSG) with Electroencephalogram (EEG) measurements) is almost infeasible to deploy at home in a “ubiquitous” manner. In addition, it is costly to train clinicians for the diagnosis of sleep conditions. In this paper, we proposed a novel technical and systematic attempt to tackle the previous barriers: first, we proposed to monitor and sense the sleep conditions using the infrared (IR) camera videos synchronized with the EEG signal; second, we proposed a novel cross-modal retrieval system termed as Cross-modal Contrastive Hashing Retrieval (CCHR) to build the relationship between EEG and IR videos, retrieving the most relevant EEG signal given an infrared video. Specifically, the CCHR is novel in the following two perspectives. Firstly, to eliminate the large cross-modal semantic gap between EEG and IR data, we designed a novel joint cross-modal representation learning strategy using a memory-enhanced hard-negative mining design under the framework of contrastive learning. Secondly, as the sleep monitoring data are large-scale (8 h long for each subject), a novel contrastive hashing module is proposed to transform the joint cross-modal features to the discriminative binary hash codes, enabling the efficient storage and inference. Extensive experiments on our collected cross-modal sleep condition dataset validated that the proposed CCHR achieves superior performances compared with existing cross-modal hashing methods.
BibTeX
@inproceedings{han2022cross,
  author = {Jianan Han and Shaoxing Zhang and Aidong Men and Qingchao Chen},
  title = {{Cross-Modal Contrastive Hashing Retrieval for Infrared Video and EEG}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=kBiOCpCCu8},
}
Figure