Semi-supervised learning trains a classifier on a few labeled images plus many unlabeled ones. It does this by letting the model label the unlabeled images itself and then training on those guesses, called pseudo-labels. The weak point is deciding which guesses to trust. Almost every method trusts a guess when the classifier's softmax confidence is high. But confidence comes from a linear readout, and that readout can be very sure about a feature that looks nothing like any real example of the predicted class. We propose a different test for each pseudo-label: can this image's feature map be rebuilt, patch by patch, using only stored patches from the predicted class? If the image is explained well by class-$c$ memory and poorly by every other class's memory, the pseudo-label is kept. If not, it is down-weighted.
The rebuilding module is a memory-augmented reconstructor, the anomaly-detection tool that forces a decoder to work only from a bank of stored "normal" prototypes. It exists only during training. It is thrown away before deployment, so the deployed network is byte-for-byte the same backbone as a plain FixMatch model. The note's claim is narrow and testable. At a matched acceptance rate, this class-conditional reconstruction margin selects cleaner pseudo-labels than softmax confidence and than nearest-prototype distance, and the difference is largest on imbalanced data and at 4 labels per class. We test whether the reconstruction step, as opposed to simply having class-indexed memory, is what earns this.
Where this sits relative to prior work
Three strands supply the pieces. Memory-augmented reconstruction comes from anomaly detection. MemAE reconstructs from a sparse, attention-addressed combination of memory slots trained on normal data, and so blocks the "identity shortcut" in which a plain autoencoder reconstructs anomalies too well. CRAD replaces discrete slots with a continuous grid and reports that such memories degrade when few samples populate them. Exploiting autoencoder's weakness to generate pseudo anomalies shrinks the reconstructible region by training against synthetic perturbations. We reuse the bottleneck idea from these papers. We change its role: it is no longer a one-class detector kept at test time, but a per-class explainer used only to score pseudo-labels. CRAD's finding that sparse data breaks memories constrains the design directly, as described below.
Training-only scaffolding supplies the deployment constraint. BatchFormer, RT-DETRv3 and DAC-DETR all attach modules that shape the backbone during training and are removed at inference. We follow the same contract. The difference is that our scaffold's main job is to decide which gradients the backbone sees, not to add a representation loss.
The closest prior work uses training-only memory against pseudo-label noise. MA-DNN keeps one running feature centroid per class plus historical predictions, and discards both at test time. DCC keeps a global feature bank for contrastive learning and denoises snippet pseudo-labels with a separate K-means stage. Neither reconstructs anything. MA-DNN's single centroid per class cannot represent a multimodal class, such as digits written in several styles or a category with several poses. DCC's clustering is a separate, non-differentiable stage over whole-video features. Our proposal differs from both in two ways. The memory stores many patch-level features per class. The trust score is a reconstruction residual restricted to one class's slots, compared against the other classes. This lets the score detect a failure a global embedding misses: an image whose pooled feature sits near class $c$ because of shared background or context, while its object patches are not explainable by class $c$.
The motivating evidence comes from the pseudo-labeling literature. FixMatch uses a fixed 0.95 confidence threshold and reports large seed-to-seed swings at 1 label per class. FlexMatch adapts thresholds per class, but documents weaker behavior on imbalanced SVHN, where low per-class thresholds let noise in. Combating Noise shows pseudo-label errors cluster near decision boundaries, which is exactly where confidence is least informative.
Core Claim
Let the backbone produce patch features $z_1,\dots,z_P$ for an image, and let $M_c=\{m^{(c)}_1,\dots,m^{(c)}_{N_c}\}$ be the memory slots attributed to class $c$. The class-$c$ residual of the image is
$$r_c(x) = \operatorname{top}\text{-}k_{\,p}\;\Big\| z_p - \sum_i w^{(c)}_{p,i}\, m^{(c)}_i \Big\|_2^2 ,$$
where $w^{(c)}_{p,i}$ are MemAE-style attention weights with hard shrinkage, which zeroes small weights so that dense mixtures cannot approximate off-class patterns. The top-$k$ average is taken over the $k$ worst-explained patches, so a localized mismatch is not diluted by well-explained background.
For a pseudo-label $\hat c$, the trust score is the margin
$$s(x) = \min_{c \neq \hat c} r_c(x) - r_{\hat c}(x).$$
The unlabeled consistency loss is weighted by a monotone function of $s(x)$, normalized per class with running statistics. This per-class normalization is what should prevent the threshold collapse on minority classes that FlexMatch reports.
The claim is that $s(x)$ ranks pseudo-labels better than confidence and better than nearest-slot distance within the same memory. We also expect it to be more stable at extreme label scarcity, because patch memories hold dozens of entries per labeled image rather than one.
Research Program
The system is a FixMatch training loop plus a side branch. The side branch is the patch memory, the sparse addressing, and a light decoder trained to reconstruct detached backbone patch features from memory. Detaching the features means the scaffold scores the backbone but does not pull it toward reconstructibility. A variant without detaching is tested separately. The memory is seeded from weakly and strongly augmented labeled images. This directly addresses CRAD's data-scarcity weakness: 4 labeled images per class, augmented many times and split into patches, yield hundreds of slots.
Growth is the hard part. It is the "memory contamination" risk, in which wrong pseudo-labels write into class slots and erode the filter. We use hysteresis. An unlabeled image is read (used for training, with its weight) at a permissive margin, but is written into memory only at a much stricter margin. Every written slot carries a provenance tag and an age, and pseudo-label-sourced slots expire unless they are reconfirmed.
Because CIFAR, STL-10 and SVHN have true labels for the "unlabeled" pool, we can measure contamination directly. This is the fraction of slots in $M_c$ whose source image is not truly class $c$, tracked over training. That measurement is itself a contribution: no paper in our file reports memory purity under self-training.
At deployment the memory, addressing and decoder are deleted. We verify that parameter count, FLOPs and latency are identical to the FixMatch backbone.
Baselines
The naive reference point is the backbone trained on the labeled images alone. It sets the floor, and MA-DNN's report of a 61.18% error for a standard CNN on SVHN with 250 labels shows how low that floor can be.
The first real competitor is FixMatch with its fixed 0.95 threshold. Every other method, including ours, is built on its augmentation pipeline, so it is the cleanest control. The file reports 88.61% accuracy at 40 labels and 94.93% at 250 labels on CIFAR-10 (FixMatch).
The strongest existing method is FlexMatch. It targets the same failure, class-dependent pseudo-label quality, at zero extra parameters. The file gives only relative figures for it: error reductions of 13.96% over FixMatch on CIFAR-100 and 18.96% on STL-10, both at 4 labels per class (FlexMatch). No absolute FlexMatch error is reported on Lacuna for CIFAR-10, CIFAR-100, STL-10 or SVHN, so we rerun it.
MA-DNN is the closest memory-based competitor. It reports 34.51% error on CIFAR-100 and 8.83% on SVHN with 250 labels (MA-DNN).
All four methods are rerun in one codebase with the same backbone, augmentations, iteration budget, unlabeled-to-labeled ratio and three seeds. Each gets the same hyperparameter search budget, about a dozen configurations tuned on a held-out labeled split. Published numbers serve only as sanity checks on our reimplementations.
Three ablations isolate the proposal's ingredients. The first scores pseudo-labels by distance from each patch to its nearest class-$c$ slot, using the identical memory with no weighted reconstruction and no shrinkage. The second replaces patch memory with one pooled feature per image, which approximates MA-DNN's granularity with multiple prototypes. The third turns off write hysteresis.
Beating FlexMatch means lower mean error across seeds on SVHN-250, CIFAR-100 at 400 labels, and STL-10 at 40 labels, with non-overlapping seed ranges on at least the imbalanced SVHN setting. It also means a strictly better pseudo-label precision–recall curve.
Why not just…?
Why not just use confidence thresholds, fixed or curriculum-adjusted? They are free, and FlexMatch is strong. But confidence is read off a linear head, which can be sure of features far from any training example. FlexMatch itself reports degraded behavior on imbalanced SVHN. The decisive test is simple. At the same number of accepted pseudo-labels per epoch, we compare the fraction that are correct under confidence, under FlexMatch thresholds, and under $s(x)$, per class. If $s(x)$ is not more precise on minority classes, the proposal fails at its central claim.
Why not just measure distance to the nearest stored prototype? This is the real threat. Reconstruction from sparse slots is close to nearest-neighbor distance when shrinkage keeps only one slot. If the first ablation matches the full method, the honest conclusion is that class-indexed patch memory, not reconstruction, carries the gain. We would report that and simplify the method. We expect reconstruction to win on multimodal classes and on images whose error is localized, because a convex combination of a few slots covers within-class variation that one neighbor does not. The top-$k$ residual, rather than a mean, targets exactly the patches a pooled distance hides. The per-class breakdown and a split of pseudo-label errors by "object patch versus context" (using CIFAR foreground crops as a proxy) will show whether this happens.
Why not just use a plain autoencoder's error? MemAE and the pseudo-anomaly work both show unconstrained autoencoders reconstruct off-distribution inputs too well. The figures are 0.697 versus 0.712 AUC on ShanghaiTech (MemAE), and 60.10% versus 67.76% AUC on CIFAR-10 one-vs-rest (pseudo anomalies). A plain autoencoder also has no class structure, so it cannot produce a margin between classes. We include it as a scoring ablation anyway, since the anomaly-detection evidence was gathered for a different task.
Why not just ensemble heads or use dropout passes, as in Tri-net? Tri-net needs repeated stochastic passes and deliberate tricks to keep its heads diverse. Our score costs one addressing pass per image. We will report wall-clock training cost against a two-pass dropout-disagreement filter. We concede that if dropout disagreement matches our precision at similar cost, the memory adds no value.
Proposed Experiments
The main runs cover CIFAR-10 at 40 and 250 labels, CIFAR-100 at 400 labels, STL-10 at 40 labels, and SVHN at 250 and 1,000 labels. SVHN is the stress test for class imbalance.
The primary metric is test error. Equally weighted diagnostics are the pseudo-label precision–recall curve against true labels, per-class precision on the three rarest classes, the memory contamination rate over training, and seed variance at 4 labels per class.
A secondary check moves the same patch filter to semi-supervised detection on COCO with 10% labels. Region proposals are a natural fit for localized residuals there. The reference is Combating Noise at 32.43% AP, and it is run only if the classification results hold.
Two outcomes would settle the question. The first is whether the full method beats the nearest-slot ablation in pseudo-label precision on SVHN minority classes. The second is whether contamination stays below the unfiltered pseudo-label error rate throughout training when hysteresis is on, and rises toward it when hysteresis is off. If both hold and deployed cost is identical to FixMatch, the reconstruction bottleneck has earned its place as a training-only filter. If the first fails, the paper becomes a clean negative result about what memory contributes to pseudo-labeling.
References
- Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
- Continuous Memory Representation for Anomaly Detection
- Exploiting autoencoder's weakness to generate pseudo anomalies
- BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning
- RT-DETRv3: Real-Time End-to-End Object Detection with Hierarchical Dense Positive Supervision
- DAC-DETR: Divide the Attention Layers and Conquer
- Semi-supervised Deep Learning with Memory
- Exploring Denoised Cross-video Contrast for Weakly-supervised Temporal Action Localization
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
- Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
- Tri-net for Semi-Supervised Deep Learning