Many vision systems for small or faint targets run in two stages. A cheap first pass looks at a shrunken, low-resolution version of the image and decides where to look. An expensive second pass then examines only those places at full resolution and draws precise outlines. The design saves a great deal of compute, but it has a known weakness. Anything the first pass overlooks is never examined again. The first pass is also the stage that sees the image after repeated shrinking has already blurred tiny, faint targets into the background. As a result, the targets that most need the high-resolution pass are the ones least likely to be sent to it. This holds for lesions that blend into nearby tissue, flat polyps, and small pedestrians in drone footage.
We propose to retrain the first-pass decision, which we call the gate, so that it answers a different question. Today the gate is asked whether an object is present at a location. We would ask whether the coarse system is likely to get that location wrong. We then set the gate's threshold with a held-out calibration procedure so that the fraction of true targets that never receive high-resolution processing stays below a chosen level, such as 2%. The claim to test is concrete. At a fixed compute budget, a gate trained to predict where the second stage helps, with a calibrated threshold, should recover more faint and small targets and draw better boundaries around them than the current alternatives. Those alternatives are lowering the threshold of an objectness gate, or improving the backbone's downsampling.
Core Claim and Position in Prior Work
The motivating evidence is consistent across domains: once the first stage misses a target, later stages cannot recover it. Riedlinger et al. found that a post-hoc classifier on top of a LiDAR detector can remove false positives but cannot recover missed objects (Riedlinger et al.). Schwartz et al. found that rescoring detections using their neighbors cannot create a detection where the response map is empty (Schwartz et al.). Cheng et al.'s search-and-segment pipeline for CT lesions shows the same pattern from the other side. Segmenting on crops reaches 62.6% Dice, against 34.1% for nnUNet on the full slice, yet the authors acknowledge that their detector is a single point of failure (A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation). Cropping is clearly worth its cost. The open question is how to stop it from silently discarding targets.
The closest existing method is QueryDet (QueryDet). It predicts a sparse heatmap from coarse feature levels (strides 8 and 16) and runs high-resolution sparse convolutions at stride 4 only where that heatmap fires. It reports a 3.0× speedup on COCO and 2.3× on VisDrone. QueryDet's heatmap is trained on objectness: it is supervised to fire where small objects are. We keep its sparse-routing design but change both what the gate is trained to predict and how its threshold is chosen. The change matters most when targets have low contrast. In that case a faint lesion produces a weak objectness response at coarse resolution, but its features are exactly the ambiguous kind on which the coarse and fine predictions are likely to disagree. A gate trained on that disagreement can learn to flag "something faint and uncertain is here." A gate trained on objectness learns to stay quiet in that situation.
A second group of papers fixes the downsampling itself. LIP learns per-pixel importance weights before pooling and raises small-object AP in Faster R-CNN from 21.9% to 24.0% on COCO (LIP). Cheng et al. use space-to-depth convolutions, which rearrange pixels into channels instead of discarding them. These methods supply a component we can reuse, since a better backbone makes every gate better. They do not address the routing decision, and Cheng et al. still have the single point of failure even with lossless downsampling. We therefore treat backbone downsampling as an orthogonal factor and cross it with gate design in our experiments, so that it is not presented as a competing explanation.
A third group of papers constrains what the second stage may consume. Price et al. find that feeding SAM a coarse mask lowers AP@50-95 by 0.133, while a box prompt raises it by 0.320 (Mastering SAM Prompts). RecycleNet (RecycleNet) and Xia et al. (Combating Uncertainty with Novel Losses) report the opposite: feeding predictions back helps, with Dice rising from 91.54% to 92.24% in Xia et al. We take a position on this disagreement and test it. Our second stage receives full-resolution image features plus the gate's continuous uncertainty map, never a binarized coarse mask. One ablation adds the coarse mask back in, which tells us which side of the disagreement applies to this setting.
Research Program
The system has three parts.
Backbone and coarse head. A standard backbone with a feature pyramid feeds a coarse head that produces detections at strides 8 and 16.
Sparse fine head. A sparse high-resolution head at stride 4, in the style of QueryDet, produces refined boxes and masks at the locations the gate selects. It is trained with cross-entropy plus Xia et al.'s Overlap Loss, which penalizes overlap between the foreground and background probability maps and so pushes boundaries to be sharp. We use that loss because it raised Dice from 82.01% to 89.83% on the 2018 Atrial Segmentation Challenge.
Gate. The new component is the gate, a small convolutional head on the coarse features that outputs a score $g(x)\in[0,1]$ at each coarse location $x$. Its training target is the benefit of fine processing:
$$t(x)=\mathbb{1}\left[\ell_{\text{coarse}}(x)-\ell_{\text{fine}}(x)>\delta\right],$$
Here $\ell_{\text{coarse}}(x)$ and $\ell_{\text{fine}}(x)$ are the per-location detection-plus-mask losses of the coarse and fine heads against ground truth, and $\delta$ is a small margin. The target is recomputed online from the current heads. During training only, the fine head runs densely on a random subset of crops so that $\ell_{\text{fine}}$ is defined everywhere. At test time the fine head runs only where the gate fires. Locations near ground-truth boundaries are the places where the coarse mask is wrong but the fine head is right. They therefore become positive targets automatically, without any hand-designed boundary band.
Threshold calibration. The threshold is not chosen by eye. We hold out a calibration split. For each true target $i$ we record $s_i=\max_{x\in \text{target } i} g(x)$, the highest gate score anywhere on that target. We set $\tau$ to the empirical $\alpha$-quantile of the $s_i$, adjusted with the usual finite-sample correction. Under exchangeability this bounds the expected fraction of test targets that receive no fine processing:
$$\Pr\left[s_{\text{new}}<\tau\right]\le\alpha .$$
The guarantee is distribution-free, but it holds only on data like the calibration split. We report how it degrades under domain shift, for example when training on Kvasir-SEG and testing on ETIS. Kang et al. report that many models lose more than 38% there (MISNet).
Measurement. The analytic contribution is a measurement we have not seen reported in the papers we reviewed. We call it the cascade miss rate: the fraction of ground-truth targets that never receive any fine-stage computation. We break it down by target size and by boundary contrast. Boundary contrast is the mean intensity difference across a thin band straddling the ground-truth contour. This lets us check directly whether objectness gates lose disproportionately many faint targets, and whether a gate trained on expected miss closes that gap.
Baselines
Naive reference. A single-pass coarse detector with no fine stage, together with the same model given a dense stride-4 level. The dense version gives the accuracy ceiling and the compute cost. QueryDet reports that adding the dense stride-4 level drops speed from 13.6 to 4.85 FPS.
Strongest existing method on routing. QueryDet with its objectness heatmap. It is the strongest competitor because it is the one method in the file that already trades high-resolution compute against recall through a learned gate. On VisDrone it reports +2.1 mAP and +3.2 AP50 over its RetinaNet baseline at 2.3× the speed of the dense high-resolution model. On COCO it reports +1.1 mAP and +2.0 AP_S at about 14.9 FPS. The file does not give absolute VisDrone mAP values, so we will re-run QueryDet ourselves. For a fair comparison it gets the same backbone, schedule, and tuning budget, and its threshold is swept across its full range so that it traces a whole compute–recall curve. Beating it means a higher position on that curve at matched FLOPs, specifically lower cascade miss rate and higher AP_S, rather than a single operating point.
Strongest methods in the medical domains. On DeepLesion, Cheng et al.'s search-and-segment pipeline reports 70.1% mAP50 and 62.6% Dice. Li et al.'s Bounding Maps report 86.28% sensitivity at 4 false positives per image with Faster R-CNN and 90.71% with MVP-Net (Bounding Maps for Universal Lesion Detection). On polyps, MISNet reports mDice 0.903 on Kvasir-SEG and 0.918 on CVC-ClinicDB. Here beating the baseline means matching or exceeding its Dice or sensitivity while lowering the cascade miss rate on the lowest-contrast third of lesions.
Ablations isolating the key ingredient. Three ablations, each changing one thing. First, the same architecture with the gate trained on objectness instead of $t(x)$, with the calibrated threshold kept. Second, the gate trained on $t(x)$ but with a hand-set threshold. Third, a LIP backbone replacing the standard one, crossed with both gate types.
Why not just…?
Why not just lower QueryDet's threshold? This is the real threat. If an objectness gate at a lower threshold reaches the same miss rate at the same FLOPs, our idea fails, and the threshold sweep described above is designed to show exactly that. Our expectation that it will fall short rests on a mechanism we have not yet verified. Lowering an objectness threshold spends extra compute evenly across background texture. A benefit-trained gate spends it where the coarse head is wrong. The contrast-stratified miss rate shows directly whether the extra compute reaches faint targets.
Why not just fix downsampling with LIP or space-to-depth convolutions? These methods help, as LIP's small-object gain shows. But Cheng et al. use lossless space-to-depth downsampling and still describe the detector as a single point of failure. The crossed design measures whether better downsampling and a better gate add up. If the LIP backbone with an objectness gate matches our gate on a standard backbone, then we would recommend the simpler LIP fix.
Why not just run a dense full-resolution segmenter? On DeepLesion, full-slice nnUNet reaches 34.1% Dice against 62.6% for crop-based segmentation (Cheng et al.). DeepStrip (Deepstrip) reports about 17 GB of memory for dense processing of 10k-pixel images, against 3.3 GB for localized processing. Dense processing is therefore both less accurate on tiny targets and too expensive at scale. We still include the dense stride-4 model as an accuracy ceiling.
Why not just prompt SAM with the first-stage boxes? The +0.320 AP gain from box prompts that Price et al. report depends entirely on the detector having proposed a box in the first place. SAM refines what it is given and has no way to recover what was missed. We include SAM with box prompts as a second-stage variant on top of each gate. If our gate's benefit holds with SAM as the refiner, the result is independent of which refiner is used.
Proposed Experiments
We will run on three benchmarks. VisDrone tests routing on tiny natural targets and gives a direct comparison with QueryDet. DeepLesion tests faint CT lesions, measured by sensitivity at 4 false positives per image, mAP50, and Dice. Kvasir-SEG and CVC-ClinicDB test low-contrast boundaries, with ETIS held out to test calibration under domain shift. Every method is reported on a compute–recall curve: FLOPs as a fraction of the dense stride-4 model, against cascade miss rate and AP_S or Dice. Boundary quality is reported separately with a boundary F-score.
The decisive check is the lowest-contrast tercile of targets at matched FLOPs. The idea succeeds if the benefit-trained, calibrated gate cuts that tercile's cascade miss rate substantially relative to the swept objectness gate, the overall miss rate stays at or below $\alpha$ on in-distribution test data, and Dice or AP does not fall. If the swept objectness gate matches it, the conclusion is that a better threshold, not a better training target, was all that was needed, and the calibration step alone becomes the contribution. Separately, the coarse-mask input ablation shows which side of the coarse-mask disagreement holds when the second stage sees full-resolution features. If adding the mask hurts, the result agrees with the SAM finding from Price et al. If it helps, it agrees with RecycleNet and Xia et al.
References
- LMD: Light-Weight Prediction Quality Estimation for Object Detection in Lidar Point Clouds
- Local Response Context Applied to Pedestrian Detection
- A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation
- QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection
- LIP: Local Importance-Based Pooling
- Mastering SAM Prompts: A Large-Scale Empirical Study in Segmentation Refinement for Scientific Imaging
- Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement
- Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Bounding Maps for Universal Lesion Detection
- Deepstrip: High-Resolution Boundary Refinement