Reconstruction-based anomaly detectors learn to redraw normal images. At test time they are asked to "heal" a defective part or a diseased scan back to a normal-looking version, and the difference between input and reconstruction marks the defect. The method rests on one fragile assumption: the training images must all be normal. Real collections from factory lines or hospital archives are rarely clean. When a few defective images slip in, the model learns to redraw the defects too, and at test time the difference map goes blank exactly where it matters. We propose to borrow co-teaching from the noisy-label literature and apply it inside the image. Co-teaching trains two networks side by side, and each one decides which training examples its peer is allowed to learn from. In our version, two reconstruction networks train together, and each hides from its peer's loss the image patches that it cannot reconstruct well. A scratch on an otherwise clean part is therefore removed from training, while the 95% of that image that is normal is kept.
The opportunity is real because the two ingredients fit together. Co-teaching works on classifiers because networks learn consistent patterns before they memorize noise, and because two differently initialized networks make partly independent early mistakes (Co-teaching). Reconstruction models show the same timing: once the loss on normal content levels off, large residuals from contaminants start to dominate the gradients and bend the model toward them (Beggel et al.). That leaves a window in which a model's residuals can be trusted to point at contamination. The open question, and the one this proposal is built to answer, is whether a peer's residual is a better filter than a model's own residual when both are applied patch by patch.
Core Claim
We claim that robustness to contamination for reconstruction detectors has to be won at the patch level, not the image level, and that cross-masking between two peers beats self-masking by a single network, with the advantage growing as contamination rises. Image-level filtering fails for subtle defects. A small scratch barely raises the mean reconstruction error of a whole image, so small-loss selection over images keeps the defective image, and the model learns the scratch. Filtering patches solves that problem, but it brings back the confirmation bias that co-teaching was invented to fix. A single network that happens to redraw a defect early will never mask it, and a single network that struggles with a complex but normal texture will mask that texture forever. Letting network A decide which patches enter network B's loss, and the reverse, breaks that loop, provided the two networks' early errors are not identical. We treat that proviso as something to measure, not assume.
The second part of the claim concerns agreement. In Co-teaching+, the two classifiers update only on examples where they disagree, which keeps them from converging to the same answers. For reconstruction we expect the opposite rule to be correct. Two models trained on normal data should redraw normal patches in nearly the same way and diverge on patterns neither has learned well. Disagreement between the two reconstructions is therefore evidence against a patch being normal. A patch is used to train peer B only if A reconstructs it well and the two reconstructions agree. The masking threshold is set per batch from a robust spread statistic over patch residuals, so there is no need to know the contamination rate in advance. That prior-rate requirement is a known weakness of both co-teaching and LOE.
Relation to Prior Work
Existing work on contaminated anomaly detection motivates this proposal but leaves the reconstruction case open. Latent Outlier Exposure treats each training point's normal-or-anomalous status as a hidden variable, estimates it with one shared feature extractor, and relabels the top $\alpha$ fraction. It needs the contamination ratio $\alpha$ as an input, works on whole samples, and was evaluated mainly with transformation-learning backbones rather than reconstruction models. We differ on all three points: two peers instead of one extractor, patches instead of samples, and a threshold with no prior. Berg et al. make a GAN-based reconstructor robust by training the encoder at the same time as the generator, which pushes contaminants toward the origin of the latent space. That result depends on a specific progressive-GAN geometry and involves no sample selection. We reuse its benchmarks as comparison points, not its mechanism.
The only paper that uses the name co-teaching for anomaly detection, CCC-T, means something different by it. It builds an appearance stream and an optical-flow stream that stay consistent through a shared memory of normal patterns. No samples are filtered, contamination is never tested, and an anomaly that shows up in both streams would be stored as normal.
Two further lines of work supply parts of the method. DRÆM provides the reconstructive U-Net and loss we build on, a U-Net autoencoder trained with $L_2$ and SSIM losses. Removing Anomalies as Noises provides the diffusion variant. Both assume the base training images are clean, and that is exactly the assumption we relax. Chadwick et al. showed that co-teaching can drop parts of an image (individual bounding boxes) rather than whole images. That is the closest precedent for spatial selection, but it was done for supervised detection, not reconstruction. Finally, Unsupervised Two-Stage Anomaly Detection uses two networks in series to trade off coarse against fine reconstruction. Its networks do not audit each other, and it too assumes clean training data.
Research Program
We will build two copies of the same reconstruction network, A and B, with different random initializations and independently drawn augmentations. Both see every mini-batch. Each image is divided into patches $p$, and each network produces a per-patch reconstruction error, $r_A(p)$ and $r_B(p)$. It also produces a peer-disagreement score $\delta(p)$, the squared difference between the two reconstructions over the patch. Network B's training loss counts a patch only when A certifies it:
$$ \mathcal{L}_B = \sum_{p} m_A(p)\, \ell_B(p), \qquad m_A(p) = \mathbb{1}\big[r_A(p) < \mathrm{med}(r_A) + k\,\mathrm{MAD}(r_A)\big]\cdot \mathbb{1}\big[\delta(p) < \mathrm{med}(\delta) + k\,\mathrm{MAD}(\delta)\big]. $$
Here $\ell_B(p)$ is B's reconstruction loss on patch $p$, $\mathrm{med}$ and $\mathrm{MAD}$ are the median and median absolute deviation over all patches in the batch, and $k$ is a single tolerance. Network A's loss is defined symmetrically. After a short warm-up with no masking, $k$ is annealed from loose to tight. This replaces co-teaching's drop-rate schedule, which depends on knowing the noise rate. At inference we deploy one peer, or both averaged, so the cost at test time matches a single detector.
The main instance uses the DRÆM reconstructive U-Net, because pixel-level ground truth for MVTec contaminants makes the mechanism directly measurable. The second instance applies the same masks to the per-patch noise-prediction loss of a diffusion reconstructor in the style of Removing Anomalies as Noises. This tests whether the idea survives when the "reconstruction loss" is a denoising objective.
The analysis has three parts. First, we will track over training the precision and recall of each peer's mask against the true defect pixels of the contaminating images. Second, we will track the overlap between A's masked set and B's masked set, which is the empirical test of whether the peers make independent errors. Third, we will record the epoch at which blind training's residuals on contaminants collapse, to locate the early-learning window for reconstruction. If the two peers' masks overlap almost completely, cross-masking reduces to self-masking. In that case, the fallback is to make the peers structurally diverse by giving each a different input corruption, such as different random patch masks, and to report whether that restores independence.
Baselines
The naive reference point is blind training: the same U-Net trained on the contaminated set with no filtering. It is paired with a clean oracle trained on the same set with the contaminants removed. Every other method is scored by how much of the gap between these two it recovers.
The strongest existing method on clean data is DRÆM itself, which reports image-level AUROC of 98.0 and pixel Average Precision of 68.4 on clean MVTec AD. We will retrain it with its synthetic-anomaly pipeline on our contaminated splits. It is the strongest competitor because its discriminative head might partly compensate for a corrupted reconstructor. Our method beats it if, at 10% contamination, our peer-masked version (with the same DRÆM discriminative head attached) exceeds contaminated DRÆM in pixel AP by more than three seed standard deviations. We found no reported result on Lacuna for DRÆM or any reconstruction detector on contaminated MVTec AD, so every number in this regime will be produced by us under identical conditions.
The strongest existing method for contamination is LOE. We will run it in two forms. The native form uses its transformation-learning backbone, which reports AUC of 94.9% on CIFAR-10 against 93.5% for its Refine baseline, and 86.8% with Soft LOE on UCSD Ped1 at 10% contamination. The adapted form uses reconstruction error as the normality loss and a hinge on reconstruction error as the anomaly loss. In both forms we give LOE the true contamination ratio, a deliberate advantage. On the GAN side, Berg et al. is the reference point on their own benchmarks, where they report AUC of 0.89 on KTH-Cellvideos and 0.64 on CIFAR-CAR at 2% contamination.
Three ablations isolate our ingredients. Patch-level self-masking uses the same threshold rule, but each network masks its own loss; this isolates the cross-update. Image-level co-teaching drops whole images by mean residual; this isolates the patch granularity. Disagreement-set training in the style of Co-teaching+ updates on patches where the peers disagree; this tests our claim that agreement, not disagreement, is the right signal for reconstruction. All methods share the architecture, data splits, number of optimization steps, augmentation, and a fixed tuning budget of twenty configurations each. Tuning is scored on a held-out contaminated validation split, never on clean labels. To match total compute against our method, which trains two networks, every single-network baseline is given twice the training steps or run as a two-seed ensemble, and we report whichever is better.
Why not just…?
Why not just have one network drop its own highest-error patches? This is the most serious threat, because it captures most of what patch-level filtering buys at half the cost. The case against it comes from classification, where single-network small-loss selection loses to co-teaching because early errors reinforce themselves (Co-teaching). The case also comes from LOE's own finding that single-model Refine lags joint modeling. Neither result was obtained for reconstruction, however. The self-masking ablation and the mask-overlap measurement settle the question directly. If self-masking matches cross-masking at every contamination level, we will report that the cross-update is unnecessary for reconstruction and that patch granularity is the real contribution.
Why not just train on synthetic defects as DRÆM does? Synthetic defects teach the network what to remove, but the reconstruction target is still the base training image. If that image contains a real scratch, the network is trained to reproduce it. Contaminated DRÆM is one of our baselines, so the experiments measure how much its discriminative head compensates. Our method is compatible with DRÆM's synthetic defects rather than opposed to them.
Why not just ensemble independently trained autoencoders? Each independently trained model meets the same gradient-dominance failure (Beggel et al.). Averaging several models that have all learned the defect does not remove it. The compute-matched two-seed ensemble baseline tests this directly, and our mask-precision curves will show whether an ensemble's averaged residual on contaminants collapses on the same schedule as a single model's.
Why not just use LOE with a reconstruction loss? LOE's relabeling operates on whole samples, so it shares the problem of image-level filtering on subtle defects. It also requires the contamination ratio. Running adapted LOE with the true ratio gives it its best case. If it still trails patch-level methods on small-defect categories such as MVTec's screw and capsule, that points to granularity as the cause.
Proposed Experiments
The main benchmark is MVTec AD with controlled contamination. Defective test images, together with their ground-truth masks, are moved into each category's training set at rates of 0%, 2%, 5%, 10%, and 20% of training images, capped by how many defects each category has. Evaluation uses the remaining held-out test images. We report image AUROC, pixel AP, and Per-Region Overlap (the average fraction of each true defect region that the detector covers), over five seeds. For context on clean data, Removing Anomalies as Noises reports PRO of 94.1% on clean MVTec AD, which the diffusion instance should approach at 0% contamination. Secondary benchmarks are KTH-Cellvideos and CIFAR-CAR at 2% contamination, for direct comparison with Berg et al. We also run the diffusion instance on BraTS, contaminated with tumor slices, where lesions are large enough that image-level and patch-level filtering should differ less. That is a useful control.
The decisive check has three parts. On MVTec at 10% and 20% contamination, peer cross-masking must beat patch self-masking in pixel AP beyond seed noise. The gap must widen with contamination. And the training-time mask recall on true defect pixels must be higher for cross-masking, with peer mask overlap well below one. A second check is that agreement-gated masking must beat disagreement-set training; if it does not, our reading of the tension with Co-teaching+ is wrong. A third check is that at 0% contamination our method must lose no more than a point of pixel AP relative to blind training, which shows that masking does not throw away hard normal texture. If the first check fails while the patch-level variants still clearly beat image-level filtering and LOE, the paper becomes a study showing that granularity, not peer auditing, is what matters for contaminated reconstruction training. That is still a useful result, and one the field currently lacks.
References
- Co-teaching: Robust training of deep neural networks with extremely noisy labels
- How does Disagreement Help Generalization against Label Corruption?
- Latent Outlier Exposure for Anomaly Detection with Contaminated Data
- Unsupervised Adversarial Learning of Anomaly Detection in the Wild
- Consistency-constrained unsupervised video anomaly detection framework based on Co-teaching
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- Removing Anomalies as Noises for Industrial Defect Localization
- Training Object Detectors With Noisy Data
- Unsupervised Two-Stage Anomaly Detection