What Respiratory Audio Models Remember When the Epidemic Moves On

Smartphone cough and breath classifiers are usually built once, on a crowdsourced snapshot, and scored with random splits. Real screening would run for months, the model would be updated as new test results arrive, and the people who choose to record would change as the epidemic moves through waves. This proposal asks a narrow question that current evaluations cannot answer. When such a model is updated over time, does it hold on to a real acoustic sign of disease, or does it hold on to shortcuts? By shortcuts we mean the symptom mix, phone models and recording rooms that happen to separate positives from negatives in a given month.

The core idea fits in two sentences. Forgetting, meaning how much accuracy on earlier data drops after an update, should be measured only on subsets where positives and negatives are matched on reported symptoms, demographics and device. On unmatched data, "remembering" can simply mean remembering a confounder. With that metric fixed, we test a specific architectural prediction from continual-learning theory: updates that barely change the pretrained encoder protect the acoustic signal better than full fine-tuning, and full fine-tuning instead rewrites deep layers toward whatever shortcut currently dominates.

Core Claim

Three findings motivate the gap. Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers showed that on the UK COVID-19 Vocal Audio Dataset, audio classifiers reach ROC-AUC 0.846 unadjusted, fall to 0.619 once positives and negatives are matched on measured confounders, and are beaten in expected utility by self-reported symptoms. That analysis is cross-sectional. On the Impact of Industrial Delays when Mitigating Distribution Drifts showed, on financial data, that ignoring the wait for labels and the wait for deployment overestimates the benefit of drift mitigation by up to 39.86%, and that a sliding window of recent data often beats keeping all history. Nobody has combined the two: chronological, delay-aware updating of an audio classifier whose labels come from passive surveillance, where who gets tested, and who records, depends on symptoms.

The architectural hypothesis comes from continual-learning theory. Wide Neural Networks Forget Less Catastrophically and The Importance of Being Lazy show that width reduces forgetting only when training stays in the "lazy" regime, where internal features barely move and task gradients stay nearly orthogonal. Measuring Representational Shifts in Continual Learning defines layer-wise representation discrepancy $D^k_t$: the smallest error left after linearly aligning layer $k$'s features before and after $t$ updates. They show it grows with depth and saturates. All of this evidence comes from vision benchmarks with artificial task boundaries.

We claim that on passively ascertained respiratory audio, the choice of forgetting metric decides the conclusion. Specifically, we predict two things:

  1. Under unmatched evaluation, full fine-tuning of a deep audio encoder looks competitive, or even better than lazy updating, because it keeps re-fitting the current shortcut.
  2. Under confounder-matched evaluation, lazy updating keeps more audio-specific signal, and the difference shows up as larger $D^k_t$ in deep layers under full fine-tuning.

If matched and unmatched rankings agree, the claim is wrong, and the field can keep using simpler metrics.

Research Program

Encoder. We use OPERA-CT, the contrastive respiratory transformer from Towards Open Respiratory Acoustic Foundation Models. That paper reports that OPERA models beat general audio encoders on 16 of 19 tasks under static splits. OPERA supplies the method; we do not propose a new encoder. Its own pretraining corpus includes UK COVID-19 and Coswara audio. We therefore run a check that retrains the encoder, or uses a released variant, with the evaluation-period audio excluded.

Data stream. Recordings are ordered by submission date and grouped into monthly periods. Labels, from PCR tests or self-report, become available only after a labeling delay, and updated models go live only after a deployment delay. We follow the protocol of On the Impact of Industrial Delays, which we reuse as a method. A temporal gap between training and test data blocks autocorrelation leakage, following Temporal Convolutional Networks for the Advance Prediction of ENSO.

Update regimes. We vary how far each update moves the encoder along a single axis:

  • a frozen encoder with a linear head;
  • a frozen encoder with a wide two-layer head, at several widths;
  • fine-tuning only the top $L$ transformer blocks, for several values of $L$;
  • full fine-tuning.

This probes the lazy-versus-rich question of The Importance of Being Lazy on an audio transformer, which that line of work has not done.

Measurements.

  • Forgetting. After each update, we measure AUROC on earlier periods, restricted to confounder-matched subsets built as in Audio-based AI classifiers show no evidence…. These subsets match symptom profile, age, sex and, where metadata allows, device class.
  • Value of audio. We report incremental AUROC of audio plus symptoms over symptoms alone, period by period.
  • Representation change. We compute $D^k_t$ at every encoder layer.
  • Shortcut probe. We track how well a linear probe on the embeddings can predict the device or recruitment-channel label, and whether this rises over updates.

Correction for who records. Passive surveillance literature corrects case counts by dividing by the chance of being observed. Examples include the symptom and care-seeking back-calculation in On the origin and timing of Zika virus introduction in Brazil and the testing-intensity probability tree in Estimating incidence of sexually transmissible infections in Australia. We carry this over as importance weights. Each training recording in period $t$ is weighted by $$w_i = \frac{1}{\hat p_t(\text{recorded, labeled} \mid s_i)},$$ where $s_i$ is the recording's stratum (reported symptoms and demographics) and $\hat p_t$ is that stratum's share of period $t$ recordings divided by its share over the whole study.

We use the pooled study distribution as the reference. Doing so assumes the population share of strata is stable, so a sensitivity analysis also uses the first period as the reference. This weighting removes shifts in the symptom mix between periods. It does not recover people who never record. Those people stay unobserved, and the note reports them as a coverage limit rather than something the model corrects.

Baselines

Frozen snapshot. This is the naive reference. It is the OPERA-CT linear probe trained on the first period and never updated. Any update scheme that fails to beat it on matched later periods is not worth deploying.

Symptom checker. This is the strongest external competitor. It is a logistic regression on self-reported symptoms and demographics, updated on the same delayed schedule. It is the strongest competitor because Audio-based AI classifiers show no evidence… found that symptoms alone beat audio classifiers once the audio classifiers were matched to 0.619 ROC-AUC. Audio beats it only if audio plus symptoms achieves a higher matched AUROC than symptoms alone in most periods, with bootstrap confidence intervals that exclude zero. We expect this bar may not be cleared. If so, the confounder-matched forgetting result still stands as a finding about evaluation.

Sliding-window retraining. This is the strongest drift-handling method in the file. The full model is retrained from the OPERA initialization on the most recent $W$ periods, with $W$ tuned on a held-out early stretch of time. It follows On the Impact of Industrial Delays, which reports that sliding windows often beat using all history.

Experience replay. This is standard fine-tuning plus a fixed-size buffer of past recordings. It represents the algorithmic continual-learning answer. Architecture Matters in Continual Learning reports that removing global average pooling matched a 1,000-sample replay buffer on Split CIFAR-100, raising accuracy from 45.0% to 67.4%. We test both components together as well as separately.

Existing results for comparison. No reported result was found on Lacuna for any respiratory audio model under chronological, delay-aware evaluation. The nearest static figures are 0.846 (unadjusted) and 0.619 (matched) ROC-AUC from Audio-based AI classifiers show no evidence…, and DiCOVA test AUCs of 69.85% (cough) and 76.85% (breathing) from DiCOVA Challenge.

Fairness. All methods see identical data, delays and label availability. Each gets the same hyperparameter search budget, measured in trials on the early held-out stretch, and GPU-hours are reported for each.

Ablations. Two ablations isolate the key ingredients. The first runs each update regime with and without the importance weights. The second evaluates the same trained models under unmatched and matched forgetting metrics, to show whether the metric alone flips the ranking.

Why not just…?

Why not just use the symptom checker? It may well win on COVID detection, and the evidence in Audio-based AI classifiers show no evidence… makes this a real threat. We treat it as the bar audio must clear, not as a straw man. The proposal remains useful if audio loses. It tells deployers whether any audio signal survives chronological updating. It also covers settings where symptoms are uninformative, since passive systems miss largely subclinical infections, as the Zika analysis shows. The encoder and protocol also carry over to lung-function estimation, where OPERA-GT gives continuous measurements that checklists cannot.

Why not just retrain on a sliding window? This is the most likely strong baseline, and it is included as one. It sidesteps forgetting by discarding the past. However, the matched-forgetting metric asks whether signal from earlier waves is lost. Our comparison checks whether the window's gains on recent periods come from tracking the current symptom and device mix, which the shortcut probe would reveal, rather than from acoustics.

Why not just use replay on a fixed architecture? Replay addresses how much is forgotten, not what is forgotten. A buffer drawn from confounded periods can replay shortcuts just as well as signal. Running replay under both metrics, and with and without importance weights, shows whether it protects matched performance.

Why not just match on confounders once, statically? That is the existing analysis. It cannot show how confounding changes as the recording population shifts between waves, or how update choices interact with that shift. Our first-period frozen snapshot evaluated on matched later periods reproduces the static view, so the added value of the temporal analysis is directly visible.

Decisive Checks

The main dataset is the UK COVID-19 Vocal Audio Dataset: 67,842 participants with PCR results. Coswara and DiCOVA serve as a second stream with a different recording pipeline. The protocol depends on recording dates being available in the released metadata. If they are too coarse, the stream is limited to the granularity available, and this is reported rather than simulated. SRC4VC is used only to build a controlled device-shift probe. The same speech is re-recorded across smartphones, which lets us test whether embedding drift under full fine-tuning follows device identity.

The claim is supported if three things hold:

  1. The ranking of update regimes under unmatched forgetting reverses or substantially compresses under matched forgetting.
  2. Lazy regimes show lower deep-layer $D^k_t$ and a flatter shortcut-probe curve.
  3. The importance weights reduce the gap between matched and unmatched AUROC.

The claim is refuted if matched and unmatched rankings agree across both streams. In that case, deep audio encoders behave like the vision results predict, and the added evaluation cost is unnecessary.