A home sleep monitor usually records only heart rhythm and breathing. The hospital test records brain waves as well, and brain waves are what sleep stages are defined by. Recent work shows that training sensors together, even when only one will be used later, makes that one sensor's encoder better. Every such method, however, aligns sensors one 30-second slice at a time. Sleep is a sequence with a grammar: light sleep (N1) and the onset of dreaming sleep (REM) are recognized mostly by what came before and what comes next. That grammar is exactly what a heart-and-breathing model is worst at reading on its own.
We propose to transfer that grammar across modalities. A cheap-sensor model reads each 30-second slice with patch self-attention, meaning the signal is cut into short chunks that attend to one another. It then carries a running memory across slices with a causal linear recurrence, a memory vector updated once per slice using only the past. We train that running memory to predict what the brain-wave encoder will say about the current slice and the next few, using unlabeled overnight recordings. At deployment the brain-wave branch is thrown away. What remains is a streaming heart-and-breathing model whose memory has learned to anticipate brain-defined transitions.
Core Claim
Cross-modal transfer in sleep is limited because alignment targets snapshots rather than dynamics. Making the cheap sensor's running memory predict the expensive sensor's current and future epoch embeddings should improve cardiorespiratory-only staging. The gains should be concentrated at stage transitions, and the model should still run online.
This differs from the closest work in three ways. sleep2vec aligns up to nine biosignals with a patch Transformer (RoFormer) and metadata-aware contrastive negatives. It reports 83.0% SHHS staging accuracy from heart and breathing alone, against 88.6% with all channels. It treats epochs independently and has no model of the overnight sequence. SleepFM contrasts each modality against the average of the others, again per epoch, and classifies with a per-epoch linear head. SeqSleepNet motivates our focus. Its bidirectional GRU over sequences of epochs raised N1 sensitivity by 17.2% absolute over per-epoch models, which shows that the grammar matters. But it is supervised, uses brain signals at test time, and looks at future epochs. We keep its insight and change both the supervision (label-free, cross-modal) and the direction of time (causal).
Research Program
The model has three parts. Each sensor family (ECG-derived inter-beat intervals, chest/abdomen respiration, SpO₂) gets a patch encoder within each 30-second epoch. Each channel is tokenized separately into short patches with shared weights, following PatchTST, and the results are pooled into an epoch embedding $z_t$.
Across epochs, a diagonal linear recurrence carries the memory. We borrow the stable parameterization of BLUR: eigenvalues are kept strictly inside the unit circle, and training uses a parallel scan. We keep only its forward scan. BLUR's bidirectional form cannot stream, and streaming is the deployment setting we care about. A separate patch encoder for EEG/EOG/EMG produces target embeddings $e_t$, and it exists only during pretraining.
The objective is
$$ h_t = \lambda \odot h_{t-1} + W z_t,\qquad \mathcal{L} = \sum_{k=0}^{K} \mathrm{InfoNCE}\big(g_k(h_t),\ \bar e_{t+k}\big) $$
The symbols are defined as follows. $h_t$ is the running memory after epoch $t$. $\lambda$ is a learned per-dimension decay with $|\lambda|<1$. $W$ projects the cheap-sensor epoch embedding $z_t$. $g_k$ is a small head predicting $k$ epochs ahead, up to horizon $K$ (we use about 4, or two minutes). $\bar e_{t+k}$ is the brain-signal embedding from a slowly updated (moving-average) copy of the target encoder. InfoNCE is the standard contrastive loss that scores the true target against negatives drawn from other epochs. We reuse sleep2vec's demography- and site-aware negative weighting so that the memory cannot win by recognizing the recording site. A small symmetric loss also keeps the target encoder aligned with per-epoch cheap-sensor embeddings, so the brain-wave targets remain informative.
Pretraining uses the public National Sleep Research Resource cohorts that sleep2vec pooled: SHHS, MESA, MrOS, and WSC. SHHS test nights and all of APPLES are held out.
The analysis asks where the gain comes from. We split test epochs into transition epochs (within two epochs of a stage change) and stable epochs, and report accuracy and per-class F1 on each group. We also probe the memory for time since sleep onset and for cycle phase.
Baselines
The naive reference is a supervised model on cardiorespiratory signals only. It uses the same patch encoder and causal recurrence with random initialization and is trained end-to-end on SHHS labels. SleepFM shows supervised baselines collapsing when labels are scarce, so we report this at full and at few-patient label budgets.
The strongest published competitor is sleep2vec restricted to heart and breathing: 83.0% accuracy on SHHS, as reported in sleep2vec. Its cross-cohort figure is 78.4% zero-shot on APPLES, against 71.4% for SleepFM, from the same paper. The file does not say which channels that APPLES figure used, so we treat it as an all-channel ceiling, not a matched comparison. To be fair we do not rely on these numbers alone. We retrain a sleep2vec-style per-epoch contrastive model on our exact pretraining corpus, with the same parameter count and GPU hours, and then add a supervised causal recurrence head on its frozen embeddings. This "aligned snapshots plus sequence head" model is the real competitor. It has every ingredient except dynamic alignment. Beating it means at least one point of overall accuracy and, more importantly, a clear gain on transition-epoch F1 for N1 and REM, consistent across three seeds.
Two ablations isolate the key ingredient. The first sets $K=0$, so the memory predicts only the current brain embedding: is anticipation needed, or just context? The second removes the recurrence and predicts from $z_t$ alone, which recovers instance-level alignment. As a brain-signal upper reference, we report SeqSleepNet's 87.1% accuracy and κ = 0.815 on MASS from SeqSleepNet. We also train our architecture with EEG at test time on SHHS against sleep2vec's 88.6%. Neither is a target for the cheap-sensor model.
Why not just…?
Why not just put a sequence model on top of frozen sleep2vec or SleepFM embeddings? This is the most likely objection. It is exactly our strongest baseline, so the experiment settles it directly. Our bet is that labels alone are too thin a signal to teach a supervised sequence head what a cardiorespiratory precursor of REM looks like. Dense brain-embedding targets on tens of thousands of unlabeled nights should teach this better. If the frozen-embedding-plus-head baseline matches us on transition epochs, the claim is wrong, and we will report it as such.
Why not just distill an EEG staging model into the cardiorespiratory model with pseudo-labels? This is a real threat. A teacher trained on labeled SHHS can pseudo-label every unlabeled night, and a causal student can learn the sequence from those labels. We run it with the same student and the same unlabeled corpus. Our hypothesis is that a five-class pseudo-label discards graded within-stage information that an embedding keeps, such as arousal depth and spindle density. If so, our representation should win most clearly on tasks the teacher was not trained for, namely sleep-disordered breathing and age prediction, where SleepFM reports AUROC 0.941 and 0.883 from its multimodal embeddings. If distillation ties us on staging and on these transfer tasks, the label-free framing has no practical advantage.
Why not just run a full-night bidirectional Transformer over all cardiorespiratory patches? Cross-modal sleep Transformers reach up to 87.7% on SHHS with EEG and EOG using sequence attention. An offline, look-ahead model is a legitimate alternative for scoring after the night is over. We include a bidirectional variant of our own model to measure how much accuracy causality costs. Our claim is limited to the online setting. Quadratic attention over roughly 1,000 epochs times many patches is also the cost PatchTST and BLUR were built to avoid.
Proposed Experiments
The decisive test is SHHS staging with heart and breathing only, run causally. We compare against the matched aligned-snapshots-plus-recurrence baseline, with the split into transition and stable epochs reported up front. The idea succeeds only if the gain sits mainly in transition epochs; a uniform small gain would suggest extra capacity rather than transferred grammar.
The second test is zero-shot transfer to APPLES with the pretrained memory and a head trained on SHHS. This checks that site-aware negatives still block shortcuts once memory spans the night.
The third test is for the fused deployment. We drop EEG mid-night on SHHS test recordings and measure how quickly accuracy recovers to the cardiorespiratory-only level. This asks whether a memory trained to anticipate brain signals degrades more gracefully than sleep2vec's static channel gating.
A shuffled-epoch control re-runs pretraining with epoch order permuted within nights. If the transition gains survive shuffling, they do not come from learned dynamics, and the core claim fails.
References
- sleep2vec: Unified Cross-Modal Alignment for Heterogeneous Nocturnal Biosignals
- SleepFM: Multi-modal Representation Learning for Sleep across ECG, EEG and Respiratory Signals
- SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging
- A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
- Bidirectional linear recurrent models for sequence-level multisource fusion
- Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers