Remembering Past Mistakes: Cross-Epoch Hard Negative Replay for Sparse Two-Stage Detectors

Modern sparse detectors such as Sparse R-CNN look at only about 100 to 300 candidate boxes per image. Each box is a learned guess that a small network refines over several rounds. This makes them fast and removes the need for duplicate-removal post-processing. It also removes something older detectors relied on: a large pool of candidate regions from which to pick out the background patches the model finds most confusing and train on them harder. That practice, hard negative mining, was central to classical face and pedestrian detectors. Online hard example mining later carried it into deep two-stage detectors. All of these versions assume thousands of candidate regions per image. A sparse detector has no such pool. At any moment it only sees the few background regions its current proposals happen to land on, and it forgets the places it once mistook for objects.

We propose to give sparse two-stage detectors a memory of their own false alarms. During training, each image keeps a small bank of boxes where the detector previously fired with high confidence on background. On later visits to that image, these boxes are replayed as extra training-only proposals and supervised as "no object." Inference is unchanged and costs nothing extra. The negatives are real regions the model itself confused, not jittered copies of ground-truth boxes. A label-validity gate keeps unannotated true objects out of the bank. The whole mechanism rests on one empirical premise: that a sparse detector's false positives recur across epochs on regions its current proposals no longer visit. That premise can be measured before anything is built, and the first experiment does exactly that.

Core Claim

Sparse R-CNN (Sparse R-CNN) trains its learned proposals with a one-to-one assignment to ground truth, called bipartite or Hungarian matching. Every unmatched proposal gets a uniform background loss. This already includes focal loss, which down-weights easy examples. The background signal is therefore hard-weighted but local in time: it only covers regions the current proposals visit. Our claim is that a cheap per-image bank of past high-confidence false positives, replayed as masked auxiliary proposals in the later refinement stages, lowers the rate of high-confidence false positives at test time. We expect gains on COCO concentrated in precision at high recall and in fewer confident background detections, at zero inference cost.

This differs from the three nearest mechanisms. Online hard example mining (OHEM) selects the highest-loss regions from about 2000 regions per image within one iteration. We mine across iterations from the model's own past outputs, so no dense pool is needed. DINO's contrastive denoising (DINO) feeds the decoder synthetic negatives made by pushing ground-truth boxes outward. Those boxes always sit next to real objects. Ours can sit anywhere the model was fooled, such as textures, reflections, or object-like clutter. Rank-DETR (Rank-DETR) sharpens which near-miss queries count as background through a stricter matching cost. It reshapes the assignment of existing queries and adds no new negative regions.

Research Program

The bank. For each training image, we store up to $K$ (around 16) boxes. A box enters the bank when a final-stage prediction from a recent epoch scored above a threshold $\tau$ for some class and overlapped every ground-truth box of that class at IoU below 0.4. Boxes are de-duplicated against each other at IoU 0.7. Each entry stores only coordinates, the predicted class, and the epoch it was added, so the bank for COCO train2017 fits in tens of megabytes. Entries expire after a few epochs unless the model fires on them again. Filling the bank costs nothing extra, because the predictions already come out of the forward pass. This is classical bootstrapping, as in Human Face Detection in Visual Scenes and Pedestrian Detection with Unsupervised Multi-stage Feature Learning, restated for a detector that cannot enumerate regions.

Replay. When an image is revisited, we inject its bank entries into refinement stage $s$ and later stages, starting from about stage 4 of 6. Each entry enters as a box plus a shared learned "probe" feature vector. Its region features are re-cropped from the current backbone with RoIAlign. A self-attention mask keeps main proposals from attending to probes, so the main set's predictions and matching are untouched; this is the same isolation trick DINO uses for its denoising groups. Probes receive only a classification loss with an all-background target. They get no box-regression loss, and probe boxes carry no gradient.

Why late stages, and why a warm-up. Dynamic Sparse R-CNN (Dynamic Sparse R-CNN) found that early stages need looser assignment, and Rank-DETR found strict negative pressure harmful early in training. Both point the same way. Early boxes are coarse, and penalizing them risks mislabelling immature true positives. So replay begins only after a warm-up of roughly a third of the schedule, and only at late stages. We treat both the starting stage and the warm-up length as ablation axes rather than fixed choices.

The label-validity gate. A false positive mined on an incompletely labelled image may be a real, unannotated object. On COCO, which is close to exhaustively labelled, we apply a simple stability filter: an entry must fire in two separate epochs before it is replayed. On LVIS (LVIS), each image lists categories confirmed absent. A mined detection of class $c$ is replayed as a negative for $c$ only if $c$ is on that list. Otherwise it is discarded as unverifiable. This makes the label-noise problem testable rather than a caveat.

Diagnostic first. Before building replay, we train standard Sparse R-CNN and save predictions on a fixed 5k-image subset of train2017 at every epoch. We then measure two things. The first is the fraction of final-epoch high-confidence false positives that had already appeared at the same location in earlier epochs. The second is the fraction of those recurring regions that had no main proposal covering them in the epochs in between, meaning they received no background gradient. If recurrence is low, or nearly every recurring false positive was already being visited and penalized, the premise fails and we stop.

Baselines

The naive reference is Sparse R-CNN with ResNet-50, 300 proposals, and the 36-epoch schedule. The paper reports 45.0 AP on COCO 2017 (Sparse R-CNN). We reproduce it in our codebase with the same augmentation and seeds, and all deltas are measured against our reproduction. It already uses focal loss, so it doubles as the "just reweight the loss" baseline.

The strongest existing sparse two-stage method is Dynamic Sparse R-CNN. It reports 47.2% AP with ResNet-50 at 36 epochs and 47.8% with ResNet-101, at a training cost of 37 hours versus 29 on 4 A100s (Dynamic Sparse R-CNN). Our method changes only training supervision and should be orthogonal to it. The fair test is therefore to add replay on top of Dynamic Sparse R-CNN with an identical schedule. Beating it means a gain of at least 0.7 AP over our reproduction, consistent across three seeds, with no inference-cost change.

The most direct competitor for the negative-exposure question is a port of DINO's contrastive denoising into Sparse R-CNN. In this port, ground-truth boxes are pushed outward into probe groups, isolated with the same attention mask, and injected at the same stages with the same probe budget. DINO's own 49.4 AP at 12 epochs comes from a different, deformable-attention architecture and is not comparable. No reported result found on Lacuna for contrastive denoising inside Sparse R-CNN, so we run it ourselves with the same tuning budget: a sweep over the noise scale equal in size to our sweep over $\tau$ and $K$.

Two ablations isolate the key ingredient, which is that the negatives are mined. The first replays the same number of random past predictions with matched box statistics instead of high-confidence false positives. The second keeps the replay machinery but refills the bank only from the current iteration. That makes it an in-batch hard-example ranking over the 300 proposals across 6 stages, close to OHEM restricted to the sparse pool.

Why not just…?

Why not just rely on focal loss? Sparse R-CNN already trains with it, so focal loss is the baseline, not an alternative. Focal loss reweights examples the model currently sees. It cannot apply gradient to a region no proposal visits. DINO's gains over denoising baselines that also used focal loss suggest explicit negative exposure adds something beyond reweighting. The diagnostic's second measurement tests directly whether unvisited recurring false positives exist.

Why not just port DINO's contrastive denoising? This is the real threat. Many confident false positives in sparse detectors may be duplicates or poorly localized boxes near true objects, and synthetic outward-pushed boxes cover exactly that case. We run the port at matched budget. We then split false positives by type: near a ground-truth box of the same class, near a box of another class, or isolated background. Our method should win on isolated background. If the denoising port matches us on that category too, the mining is not worth its bookkeeping, and we will say so.

Why not just raise the number of proposals or add training-only extra proposals? More proposals give more background coverage at every step. Speed/Accuracy Trade-Offs reports that proposal count buys little accuracy for its cost in multi-stage detectors. Still, a training-only group of extra learned proposals is a cheap, credible rival. We include it with the same extra region count as our bank, at 300+16 versus 300 plus 16 random learned probes. If it matches replay, coverage rather than mining is what matters.

Why not just run OHEM on the sparse pool? The current-iteration ablation is exactly this. With only about 300 regions, all of which already receive a loss, we expect it to behave like focal loss. If it does not, that is a cheaper result worth reporting.

Proposed Experiments

The main study uses COCO 2017 (COCO) with ResNet-50, reporting AP, AP50, AP75, and AP by object scale over three seeds. It adds two diagnostic metrics. The first is the number of false positives per image scoring above 0.5. The second is the false-positive breakdown by type described above, so that any AP gain can be traced to the mechanism. ResNet-101 and the Dynamic Sparse R-CNN base test whether the gain survives a stronger model.

LVIS tests the gate. We compare replay with and without the confirmed-absent-category check on overall AP and on rare-category AP. The reference points are those reported in LVIS: Mask R-CNN at about 21% AP overall, and rare-category AP of 0.8% without and 13.4% with repeat-factor sampling (LVIS). No reported result found on Lacuna for Sparse R-CNN on LVIS, so we train that baseline with repeat-factor sampling ourselves. The gate earns its place if ungated replay lowers rare-category AP while gated replay holds or raises it.

CrowdHuman (CrowdHuman) is a secondary stress test. Its average of 22.6 people per image makes mined "false positives" more likely to be unmatched occluded people. We report AP and the log-average miss rate, and we expect the stability filter to matter most here. No reported result found on Lacuna for Sparse R-CNN on CrowdHuman.

The decisive checks, in order, are as follows. First, the recurrence diagnostic must show that a substantial share of final false positives were both recurring and unvisited; otherwise we stop. Second, replay must beat the random-replay and extra-proposal ablations on isolated-background false positives. Third, it must beat the denoising port on that same category while matching it elsewhere. Fourth, it must add measurable AP on top of Dynamic Sparse R-CNN without changing inference cost.