How Much of Digital Depression Detection Is Just Knowing Who Someone Is?

Phone and wearable data are routinely used to "detect depression." The models are trained on questionnaire scores, pooled across all participants and weeks, and scored with a single accuracy number. That number mixes two very different abilities. One is tracking when a given person's mood worsens, which is what a monitoring tool needs. The other is recognizing which people tend to report high scores. That second ability rests on stable traits, demographics, and each person's habitual way of answering questionnaires. On clinical outpatients, Ikäheimonen et al. found that 72% of the variance in repeated depression scores lies between people, not within them. A model that learns only "who" can therefore look accurate while knowing nothing about "when."

We propose a null-model audit that separates the two. A null model here means a deliberately scrambled version of the data that keeps some structure and destroys the rest, so that a model's score on real data can be compared with its scores on the scrambled copies. Network science uses this logic to ask whether clustering is real or just a side effect of some nodes having many links. We port it to longitudinal sensing. The scrambles keep each person's identity, label rate, and label autocorrelation, and they break the week-by-week alignment between behavior and self-report. We then re-evaluate published models on the GLOBEM benchmark against these scrambles and report how much of each model's accuracy survives. We call that surviving portion the model's state share.

Core Claim

Pooled accuracy on longitudinal sensing benchmarks is not interpretable without a trait-preserving, state-destroying null. We expect a large fraction of reported within-dataset accuracy to be reproducible by models that know nothing about week-to-week change. The mechanism behind that fraction also matters. Because it is identity-driven, it inherits two biases. The first is self-report bias: stable response styles such as social desirability sit in the between-person component of the label. The second is demographic bias: between-person differences track who was sampled.

This claim differs from the closest work in specific ways. Ikäheimonen et al. decompose variance with linear multilevel models and describe "correlation washout," where opposite-signed within-person correlations cancel when pooled. They do not audit machine-learning detectors, and they give no significance test for a black-box model's within-person skill. GLOBEM shows that accuracy collapses across years and datasets. It does not ask what the within-year accuracy was made of.

Research Program

The method borrows from degree-preserving randomization. The configuration model and the edge-swapping of Leskovec et al. keep every node's number of links and destroy everything else. Leskovec et al. and Rabbany et al. then express how surprising an observed statistic is as a Z-score against the scrambled ensemble. Our analog of "a node's links" is a person's label sequence: its base rate and its autocorrelation. We build three null ensembles that preserve progressively more.

The first is a within-person circular shift. Each person's label sequence is rotated by a random offset relative to their sensing sequence. This keeps the person's label rate and autocorrelation intact and destroys the alignment between behavior and label.

The second is a trait-only oracle. The predictor may use a person's mean behavior over the whole study, or their mean label in training weeks, but nothing time-varying.

The third, where demographic fields exist, is a stratum swap. Each person's sensing stream is replaced by that of a different participant from the same demographic stratum. This tests whether a model beats a demographic stereotype.

For each model we compute its score $S$ on real data and scores $S_{\text{null}}$ on each ensemble. From these we form

$$\text{state share} = \frac{S - \mu_{\text{trait}}}{S - S_{\text{chance}}},$$

where $\mu_{\text{trait}}$ is the mean score under the circular-shift null, and $S_{\text{chance}}$ is 0.5 balanced accuracy. We also report the Z-score of $S$ against each ensemble. All models are retrained on each null draw, so they get the chance to exploit whatever the null preserves.

For self-report bias, GLOBEM pairs weekly PHQ-4 with term-level BDI-II, two different depression questionnaires. We use each person's persistent disagreement between the two instruments as a proxy for instrument-specific reporting offset. We then test whether the trait-driven part of a model's predictions correlates with that offset. The prior for this comes from reports of social-desirability and measurement-noninvariance effects in Yue et al. and of state-dependent recall in Vernon et al..

For demographic bias, the audit is repeated on the MoMo-Mood clinical outpatient cohort, if access is granted. That cohort has an older, employed/unemployed mix where employment status carries β = −4.79 on PHQ-9 (Ikäheimonen et al.). The question is whether state share differs from the undergraduate-only GLOBEM cohort, which the demographic-bias literature, e.g. Goudbeek et al., warns truncates the space of human variation.

Baselines

The naive reference point is the majority-class or chance predictor, at 0.500 balanced accuracy. GLOBEM reports that cross-dataset deployment falls to roughly this level (Xu et al.).

The strongest existing detectors are the 18 algorithms benchmarked in GLOBEM (9 traditional, 9 domain-generalization) and the Reorder self-supervised method. GLOBEM reports within-dataset single-year balanced accuracy up to 0.691, and a 15.9% cross-dataset improvement for Reorder (GLOBEM). We also include the attention-based GRU autoencoder of AttentiveGRUAE, which reports depression AUC 0.74 on discovery data and 0.61 on external validation. We rerun all of these with the released GLOBEM code, splits, and hyperparameter grids, and give each null draw the same tuning budget as the real run.

The variance-decomposition baseline is a linear multilevel model with random person intercepts, following Ikäheimonen et al.. Its within-person contribution was about 2.1% marginal R² beyond demographics.

The key ablation compares the circular-shift null with a naive i.i.d. label permutation. The naive permutation destroys each person's autocorrelation, so any gap between the two nulls measures how much a naive null overstates state skill.

"Beating" a baseline has a specific meaning here. We do not propose a better detector. The audit succeeds if at least one headline model's state share differs substantially from what its pooled accuracy implies, and if the ranking of models by state share differs from the ranking by pooled accuracy. If every model's accuracy is almost entirely state-driven, the audit's central worry is refuted, and we will report that.

Why not just…?

Why not just use leave-subjects-out splits? Holding out whole people tests generalization to new individuals. It does not remove trait prediction, because a new person's average sleep or mobility can still reveal their average reporting level. A model can pass subject-wise evaluation while tracking nothing within a person. The trait-only oracle, run under subject-wise splits, measures exactly how much of the held-out score this route explains.

Why not just fit a multilevel model or compute per-person AUC? This is the real threat. Within-person metrics are the natural simple answer, and we report them. Two things are missing from them. Per-person AUC is undefined for the many participants whose label never changes. It is also optimistically biased when labels and features are both autocorrelated, and it comes with no calibrated test. A multilevel model gives the test but only for a linear model class. The circular-shift null applies to any black-box detector and supplies the reference distribution directly. If per-person AUC and state share rank models identically in our experiments, we will recommend the simpler metric and present the nulls only as its significance test.

Why not just residualize labels on demographics? Regressing out age, gender, or employment removes the demographic trait component that is measured. It leaves the unmeasured parts, such as response style and personality, which the circular-shift null removes regardless of whether they were recorded. Comparing state share before and after demographic residualization separates the two.

Proposed Experiments

Experiment 1: GLOBEM audit. We compute state share and null Z-scores for every GLOBEM baseline, Reorder, and a GRU-autoencoder model. This runs within year and cross-year, on the benchmark's own depression labels. The prediction is that a substantial part of the up-to-0.691 within-year accuracy is reproduced by the trait-preserving null.

Experiment 2: does state share predict transfer? We test whether state share measured within a year predicts how well a model survives cross-year deployment. If feature reversals after COVID-19 (Xu et al.) mainly hit identity-based shortcuts, trait-heavy models should collapse more.

Experiment 3: reporting offset. We test whether a model's trait-driven predictions correlate with the PHQ-4/BDI-II reporting offset. A positive result would mean part of the reported accuracy is learning how people answer questionnaires rather than how they feel.

Experiment 4: clinical replication. We repeat Experiments 1 and 3 on MoMo-Mood to see whether conclusions from the undergraduate-only cohort hold in clinical outpatients.

The decisive check is Experiment 1. If the leading models' accuracy clears the circular-shift null by a wide margin, identity confounding is a minor concern for this benchmark, and the remaining experiments become secondary.