Past, Present, Future: Label-Free Calibration of Domain-Adapted Sleep Stagers

Automatic sleep staging assigns one of five stages to each 30-second window of an overnight brain recording. Models trained on one hospital's data are routinely deployed at another hospital that has different sensors and different patients. Adversarial domain adaptation (training the feature extractor so that a second network, the discriminator, cannot tell which hospital a recording came from) recovers much of the lost accuracy. It is also known to make the model overconfident on the new data. Every existing fix for that overconfidence treats each window as an isolated sample and needs some outside estimate of how accurate the model is on the new site. This proposal uses a property of sleep recordings themselves to supply that estimate.

The property is this: given the true stage at time $t$, the signal before $t$, the signal at $t$, and the signal after $t$ are roughly independent. We train three separate predictors of the stage at $t$, one from each of these three views, and run them on unlabeled target nights. Under that independence, the pattern of agreements and disagreements among the three predictors is enough to recover each view's confusion matrix and the target stage frequencies, with no target labels. This is a classical result for latent-class models with three conditionally independent views. These estimates then calibrate the combined prediction directly on the target hospital, including at stage transitions, where clinically important errors cluster.

Core Claim

Our first claim is that in sequential physiological data, temporal structure makes the target model's accuracy estimable without labels. If past context $L_t$, the current epoch $C_t$, and future context $R_t$ are conditionally independent given the stage $Y_t$, then the joint distribution of the three views' predicted labels $a, b, c$ on target data factors as

$$ P_T(a,b,c) \;=\; \sum_{k=1}^{5} \pi_k \, M^{L}_{ak}\, M^{C}_{bk}\, M^{R}_{ck}, $$

where $\pi_k$ is the target frequency of stage $k$ and $M^{v}_{jk}$ is the probability that view $v$ predicts $j$ when the true stage is $k$. This three-way factorization is generically identifiable up to a relabeling of the stages, and the relabeling is fixed by matching to the source confusion matrices. Recovering $\pi$ and the three $M^v$ from unlabeled target sequences gives what TransCal and UTDC each lack. TransCal does not adjust for target label shift. UTDC depends on an external accuracy estimator, and the research file notes that such estimators break under clinical label shift.

Our second claim is that adversarial alignment helps this calibration, rather than being only the cause of the problem. The independence assumption is violated mainly by subject-level nuisance signal, such as a person's EEG amplitude or electrode impedance, which leaks into all three views. Removing that signal is exactly what the discriminator is trained to do. So adversarial training should both raise accuracy and make the three-view estimator valid, and this can be checked on labeled source data.

Relation to Prior Work

Two lines of work motivate the gap. TransCal shows that adversarial adaptation raises target accuracy while worsening calibration. It corrects this by reweighting source samples with a density ratio derived from the discriminator, and it assumes the class priors do not shift. UTDC argues that the real cause is the gap between source and target accuracy. It fits a temperature on target data using an external accuracy estimate such as ATC. Gong et al. move from one global temperature to cluster-level temperatures for unseen domains. All three work on static images or text and treat samples as i.i.d. Our method keeps UTDC's insight that accuracy must be estimated on the target. It replaces the external estimator with one built from the temporal structure of the data.

Several papers constrain the design. SPDIM shows that alignment methods fall into an "alignment trap" when patient cohorts have different stage frequencies from healthy subjects. That warns us that any calibrator assuming $P_S(Y)=P_T(Y)$ will be biased, which is why $\pi$ is estimated rather than assumed. Jagannatha finds that the marginal probabilities of structured CRF predictors become miscalibrated out of distribution. A CRF (conditional random field, a layer that scores whole label sequences using learned stage-to-stage transition costs) is therefore not a fix in itself. SleepTSP shows that aggregate scores hide failures at stage boundaries, which is why we report calibration at boundaries separately.

Other papers supply the method. ADAST provides the adversarial training recipe for EEG: a shared encoder, attention modules specific to each domain, a discriminator, and self-training on pseudo-labels. It processes each epoch independently and selects pseudo-labels using raw, uncalibrated confidence. DREAM provides the strongest sequence architecture, a Transformer followed by a CRF. It reports CRF entropy as an uncertainty score but never checks whether that score is calibrated. Our model differs from DREAM in one structural respect: the three views must not share a context encoder, or they lose their independence.

Research Program

The model has one epoch encoder, shared across views and trained adversarially against a hospital discriminator through a gradient reversal layer, following ADAST and drawing on the SLARDA and TS-TCC codebases. Three heads sit on top of it. The past head reads encoded epochs $t-k$ through $t-1$, the current head reads epoch $t$ alone, and the future head reads epochs $t+1$ through $t+k$. Each head outputs a distribution over the five stages, and all are trained with source labels.

On unlabeled target nights, we collect each window's three predicted labels into a three-way table of counts. We then estimate $\pi$ and the $M^v$ with a tensor decomposition, refined by expectation-maximization. Using soft predictions, binned by confidence, gives a more stable variant.

The calibrated target prediction is the posterior $P(Y_t \mid a,b,c)$ implied by these estimates. In a refined version, we instead fit a temperature and a per-stage bias for each head so that its soft confusion matrix on target matches the estimated $M^v$, and then combine the three heads. The same estimates feed two downstream uses. First, pseudo-labels for ADAST-style self-training are chosen by calibrated posterior rather than raw confidence. Second, the estimated $\pi$ reweights the discriminator's classes so that alignment does not pull patient data toward healthy-subject stage frequencies.

The analytical work has two parts. We will derive how errors in the independence assumption propagate into calibration error, which matters most for boundary epochs, where the past and future views legitimately disagree. We will also characterize when the five-stage problem is poorly conditioned. N1 is rare and resembles both Wake and N2, so its column of the confusion matrix may be estimated poorly when $\pi_{N1}$ is small.

Baselines

The do-nothing reference is the adversarially adapted sequence model's raw softmax. Next to it is the same model with a temperature fit on source validation data. On Office-Home, TransCal reports source temperature scaling at 16.8% ECE against 26.8% uncalibrated, but there is no reported result on Lacuna for calibration of any sleep stager under cross-site shift. These references will be measured on the same networks used by every other method.

The strongest existing calibrators are TransCal and UTDC, run per epoch on our adapted model with the same frozen features. TransCal reports 13.5% ECE on Office-Home. UTDC reports adaptive ECE (adaECE, an ECE variant whose confidence bins hold equal numbers of samples) of about 5–7 on Office-Home, against 15–30 for baselines. UTDC is the strongest competitor because it also calibrates on the target using estimated accuracy, and it differs from us only in where that estimate comes from. We run it with its ATC estimator under an identical budget: the same unlabeled target nights and a hyperparameter search of the same size.

We also include the cluster-level temperature regression of Gong et al. and DREAM's Transformer-CRF, whose CRF marginals and entropy serve as a structured-uncertainty baseline. For accuracy context, ADAST reports 74.00% average cross-domain accuracy against 63.72% for the source-only model, with 60.39 macro-F1. We will reproduce these numbers before adding calibration.

The key ablation feeds the three views from one shared Transformer, which deliberately breaks independence while keeping capacity fixed. A second ablation removes the discriminator. Together they show whether gains come from three-view identifiability and whether adversarial invariance is what makes it work. An oracle temperature fit on target labels gives the floor.

Beating the field means lower target ECE and adaECE than UTDC in most cross-site scenarios. It also means a lower error in estimated per-stage target accuracy than ATC, and the largest margins should appear on boundary epochs and on fragmented-sleep target cohorts.

Why not just…?

Why not just use UTDC with ATC on each epoch? This is the real threat. Under covariate shift alone with near-identical stage frequencies, it may match us. ATC sets a confidence threshold from the source and assumes the relation between confidence and correctness carries over to the target. Patient cohorts with less deep sleep and more awakenings violate exactly that assumption, as the label-shift findings of SPDIM indicate. The experiment settles it directly: we compare the error in estimated target accuracy and stage frequencies across target cohorts grouped by how much their stage frequencies differ from the source. If ATC is equally accurate in the high-shift groups, our contribution narrows to the boundary analysis.

Why not just re-estimate the transition matrix with Baum–Welch on the unlabeled target, using the network's outputs as emissions? This adapts the temporal prior to fragmented sleep. Baum–Welch is the standard expectation-maximization procedure for fitting hidden Markov models (HMMs), models of hidden states that switch with fixed probabilities. Its weakness is that it trusts the emission confidences, and those are what adversarial alignment inflates. In a hybrid neural HMM, the emissions are scaled posteriors rather than normalized likelihoods, so the overall likelihood cannot identify a temperature. Overconfident emissions also make predicted nights look more fragmented than they are, inflating the estimated transitions. SleepTSP additionally reports that post-hoc smoothing removes genuine short N1 bouts. We include Baum–Welch, with a source-fit temperature, as a baseline. If it matches us on boundary calibration, the three-view machinery is unnecessary.

Why not just fit separate source temperatures for boundary and stationary epochs? On the target, boundary locations are unknown and must be inferred from the model's own uncertain predictions. A source-fit boundary temperature also inherits the source's run lengths, which is the temporal label shift problem itself. We will run it with boundaries detected by the model and report it next to our method on the boundary-conditioned metric.

Proposed Experiments

The data are Sleep-EDF-20 and -78, SHHS-1 and -2 downsampled to 100 Hz, and ISRUC-Sleep subgroup 3. We use the cross-dataset transfer scenarios between EDF and SHHS from ADAST, plus EDF→ISRUC. Natural temporal label shift is created without breaking nights into fragments: target subjects are grouped by transitions per hour and by N3 fraction, and the patient-versus-healthy split follows the spirit of SPDIM.

The metrics are ECE, adaECE, Brier score, and negative log-likelihood. Each is reported overall and separately for epochs within one epoch of a true stage change versus stationary runs. We also report the error in estimated per-stage target accuracy and stage frequencies, the precision of selected pseudo-labels, and the downstream accuracy and macro-F1 after self-training.

Four checks are decisive. First, on synthetic HMM sequences where independence holds exactly, the estimator should recover the true confusion matrices, and the error should shrink roughly as one over the square root of the number of target epochs, the usual rate for moment-based estimators. Second, on labeled source data, we measure the conditional mutual information between the three heads' predictions given the true stage, with and without the discriminator. Our second claim predicts that adversarial training reduces it. If it does not, that claim fails. Third, calibration gains should grow with temporal label shift and be largest on boundary epochs, where UTDC and TransCal are expected to fare worst. Fourth, if the shared-encoder ablation calibrates as well as separate views, the independence story is wrong and the gain is coming from something else.