Continual learners often keep a small memory of past examples and replay them to avoid forgetting. When incoming labels are wrong, that memory becomes a trap. A mislabeled example stored in the buffer is rehearsed again and again, so the model memorizes it. Once the model has memorized it, its loss on that example looks as small as the loss on a correct one. Current defenses try to catch bad labels using the learner's own loss or feature geometry. Both signals depend on how a particular network memorizes and forgets, and both have been tested almost only on convolutional networks. The problem gets worse in the settings where it matters most, such as clinical signals, where only a small fraction of the stream carries expert labels and some of those labels are wrong.
We propose to separate the model that learns from the model that judges labels. A frozen, self-supervised "auditor" encoder embeds everything that arrives: the few labeled examples, the many unlabeled ones, and the buffer contents. Each labeled example is scored by how strongly its own label is predicted from its neighbors in that embedding, with its own label hidden, and the unlabeled data carry label information between those neighbors. Only examples that pass this held-out check are admitted to the buffer or kept in it. Because the auditor never trains on the labels it judges, the clean-versus-noisy gap cannot shrink through memorization. Because the check depends only on neighborhoods in an embedding, it can be applied unchanged to convolutional networks, Vision Transformers, and state-space models (sequence models built on linear state updates).
Core Claim
Our claim is that the "vanishing gap" found by May the Forgetting Be with You comes from scoring examples in-sample. Their own finding is that online learners memorize noisy labels so fast that loss-based filtering stops separating clean from noisy. Out-of-sample scoring on a frozen self-supervised representation removes this failure by construction. Holding out labels does two further things. It keeps noise detection accurate when labels are scarce, because unlabeled examples supply the neighborhood structure that few labels cannot. It also makes buffer purification portable across architecture families, whereas methods that depend on forgetting rates or cluster compactness are not.
Formally, let $z_i$ be the auditor embedding of example $i$ with given label $y_i$. Let $\hat{p}_{-i}(c \mid z_i)$ be the class distribution at node $i$ after label propagation over a sparse nearest-neighbor graph. The graph covers the buffer and a recent window of unlabeled stream data, and $i$'s own label is removed. The admission score is
$$s_i = \hat{p}_{-i}(y_i \mid z_i) - \max_{c \neq y_i} \hat{p}_{-i}(c \mid z_i),$$
which is the margin by which the neighborhood, with $i$'s label hidden, supports the label $i$ claims. In practice we compute it by cross-fitting over K folds of the labeled nodes, so the cost is one propagation per fold rather than one per example.
Relation to Prior Work
Two buffer-purification methods motivate the design and are the main points of contrast. Self-Purified Replay (Kim et al.) trains a label-free contrastive encoder and admits examples with high eigenvector centrality within their labeled class, using a mixture model on the centrality scores. It reports filtering 96% of noisy labels on MNIST with 20% symmetric noise. It also degrades on diffuse WebVision classes such as "Spiral" and "Cinema", and it cost over 119 hours on CIFAR-10 in the comparison run by Millunzi et al. Its signal asks whether an example is central within its claimed class. That question fails whenever a class is spread out, and it ignores unlabeled data entirely. Our score asks instead whether the neighborhood, labeled and unlabeled together, predicts the claimed label. That question stays meaningful for dispersed classes, as long as neighbors of other classes do not dominate.
Alternate Experience Replay (AER) (Millunzi et al.) restores the loss gap by periodically pausing replay so that noisy examples are forgotten first. Its purity-based sampling scheme adds 4.71% average accuracy over loss-based purification. AER approximates "a model that has not trained on this example" by making the learner forget. We compute that quantity directly with a scorer that never saw the label. We therefore do not depend on the assumption, tested only on CNNs, that noisy associations are unlearned faster than clean ones.
Work on test-time adaptation constrains the design more than it supplies it. SoTTA filters its buffer by confidence and enforces class balance. OBAO evicts high-entropy samples. ReTA documents an "entropy trap", in which shifted inputs receive confident wrong predictions. This rules out using the learner's confidence as the admission signal, and ReTA's alternative, a committee of text prompts, needs a vision-language model.
The label-efficiency literature supplies the auditor and the regime. Benchmarking ECG FMs shows that pretrained encoders give 3.3x to 9x label efficiency. It shows that a 3.8M-parameter state-space model (ECG-CPC) beats Transformers 25 times larger in 5 of 7 task categories. It also shows that encoders from different families learn nearly orthogonal representations, as measured by representational-similarity analysis. That last result is both a threat and an opportunity for our design. The threat is that auditor quality may vary by family. The opportunity is that an auditor from a different family may not share the learner's memorization errors.
Two further papers inform the design. ISD argues that soft neighborhood structure serves rare classes better than hard instance discrimination, which supports neighborhood-based scoring for long-tailed labels. Xu et al. warn that prototype methods tuned for compact CNN clusters may not transfer to Transformers. We treat this as a hypothesis to test for every scoring rule we compare, including ours.
Research Program
We will build a replay learner in which labels never touch the auditor. The auditor is a pretrained self-supervised encoder, kept frozen in the main condition. In an ablation, it is refreshed only with self-supervised updates on unlabeled stream data. The learner is fine-tuned on the stream and on the admitted buffer. At each step, newly labeled arrivals and the current buffer are scored with the held-out propagation margin above. We fit a two-component mixture to the scores and use it to decide admission and eviction, while keeping the class balance that SoTTA showed matters. Unlabeled arrivals are never admitted with pseudo-labels in the main condition. They only serve as graph nodes. This removes the self-confirmation loop that produces the entropy trap.
The analysis has three parts. First, we track the separability of clean and noisy labels over stream time for every scoring rule and every architecture, measured as the area under the ROC curve (AUROC) for noise detection. This tests whether our score avoids the vanishing gap. Second, we sweep the fraction of labeled examples in the stream (100%, 10%, 1%) to test whether the unlabeled nodes carry the benefit. Third, we cross learner and auditor families (CNN, Transformer, state-space model) to test portability, and to test whether auditing across families catches errors that auditing within a family misses.
Baselines
The naive reference is standard experience replay with reservoir sampling and no purification, run on the same pretrained backbone. It tells us how much damage the noise does and sets the floor.
The strongest existing competitor is AER with its purity-based sampling and task-boundary consolidation (Millunzi et al.). It is the best cost-accuracy trade-off in the file for online noisy continual learning, and it directly targets the vanishing gap. The file reports a 4.71% average accuracy gain over loss-based purification and 1.5 hours of training on Seq. CIFAR-10. We found no reported result on Lacuna for AER's absolute final accuracy on Seq. CIFAR-10, so we will reproduce it. We will run AER on every backbone in our study, starting from the same pretrained initialization, with the same buffer size, stream order, noise seeds and label fraction, and with an equal hyperparameter search budget for its forgetting schedule and ours. Beating it means higher final average accuracy and higher buffer purity at matched wall-clock time, on at least two of the three architecture families, with the margin growing as labels become scarcer.
SPR (Kim et al.) is the closest competitor built on self-supervised features. We run its centrality filter on the same frozen auditor embeddings, so the comparison isolates the scoring rule from the encoder. The benchmark reference point is its 96% noise-filtering rate on MNIST with 20% symmetric noise. A SoTTA-style filter, which admits by confidence with class balance (SoTTA), represents methods that trust the learner's confidence.
Two ablations isolate the key ingredients. The first is labeled-only held-out nearest-neighbor voting: identical to our method except that unlabeled nodes are removed from the graph. It isolates what label-scarce propagation adds. The second is in-sample scoring: our propagation score computed without hiding $i$'s own label, and with the learner's adapting encoder in place of the frozen auditor. It isolates the held-out, frozen-auditor ingredient that we claim removes the vanishing gap.
Why not just…?
Why not just run AER on a Transformer or state-space backbone? It may work, and it is the main threat. Nothing in the file shows that its forgetting asymmetry fails outside CNNs. It has simply never been tested there. Our clean-versus-noisy AUROC curves over stream time, measured per architecture, settle this. If AER's gap re-opens as reliably on Vision Transformers and ECG state-space models as on CNNs, and matches our accuracy, then our portability claim fails. What would remain is only the label-scarce advantage, which AER was never designed for because it assumes a fully labeled stream.
Why not just keep confident predictions from a large foundation model? ReTA shows that confident errors under shift poison caches even for CLIP-scale models. SoTTA reports that TENT and CoTTA drop to 52.1% and 54.8% on CIFAR10-C with noisy streams, against 80.0% with robust buffering. Benchmarking ECG FMs shows that scale alone does not win on physiological signals. We include the confidence filter on our largest backbones so that this is measured directly rather than argued.
Why not just do nearest-neighbor label voting on frozen features? This is the simplest version of our idea, and we expect it to work well when most of the stream is labeled. That is why it is our first ablation rather than a straw man. Our claim is narrower: at 10% and 1% labels, a labeled example has too few labeled neighbors for voting to be reliable, and propagation through unlabeled nodes should recover detection accuracy. If labeled-only voting matches propagation at 1% labels, we will report that the contribution reduces to held-out scoring on a frozen auditor.
Why not just freeze the backbone and fit a noise-robust head, with no buffer purification? A frozen encoder cannot memorize noise in its features, but its head still can. A frozen encoder also gives up the adaptation gains that ECG fine-tuning shows over frozen probing. We include a frozen backbone with a nearest-class-mean head, trained on all received labels, as a reference point. Our method must beat it to justify fine-tuning at all.
Proposed Experiments
The vision track uses Seq. CIFAR-10 and Seq. CIFAR-100 with 20%, 40% and 60% symmetric and asymmetric noise, following Kim et al. and Millunzi et al.. It also uses Seq. Food-101N and WebVision for real annotator noise, with a per-class analysis of the abstract classes where SPR failed. Each benchmark is run with 100%, 10% and 1% of stream examples labeled, and with ResNet and ViT learners and auditors.
The clinical track turns the 12-lead ECG suite from Benchmarking ECG FMs into a class-incremental stream over its classification tasks, with injected label noise and the same label fractions. It uses the pretrained ECGFounder (CNN), ECG-JEPA and HuBERT-ECG (Transformers), and ECG-CPC (state-space model) as both learners and auditors. This gives a 3x3 grid of learner and auditor families on identical data, and it lets us check whether ECG-CPC's advantage as a representation carries over to auditing.
The metrics are final average accuracy, buffer purity over time, clean-versus-noisy AUROC over stream time, and wall-clock time.
Three results would decide the proposal. First, our AUROC should stay flat over the stream while in-sample loss scoring decays; if it decays too, the held-out argument is wrong. Second, the gap between propagation and labeled-only voting should widen as the label fraction drops from 10% to 1%. Third, the ranking of methods should be stable across the three architecture families, and at least one pair of learner and auditor from different families should beat the same-family pairing on purity. If only the first result holds, the contribution is a cheap, architecture-agnostic replacement for AER's forgetting schedule. If all three hold, frozen held-out auditing becomes a general design principle for label-scarce continual learning on noisy streams.
References
- May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels
- Continual Learning on Noisy Data Streams via Self-Purified Replay
- SoTTA: Robust Test-Time Adaptation on Noisy Data Streams
- Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation
- Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- ISD: Self-Supervised Learning by Iterative Similarity Distillation
- Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization