Crowd counts feed decisions where a wrong number is costly, such as stadium gate control, evacuation planning, and social-distancing enforcement. Yet every counting network in current use outputs a single number for every image, with no way to say "I cannot count this part." This proposal builds a counter that can hand specific regions of an image to a human while counting the rest itself. It comes with a statistical guarantee: across new images, the error in the final human-plus-machine count stays below a budget the user chooses.
The signal that decides which regions to hand off comes from hybrid counting, meaning models that run an object detector and a density regressor side by side. The detector draws boxes around individual heads. The regressor predicts a smooth "people per pixel" surface whose integral is the count. The two are known to fail in opposite places: the detector undercounts in congested areas, and the regressor hallucinates people in foliage and stonework. Where the two counts disagree on a patch, at least one of them is wrong. We treat that disagreement as the deferral score. We then calibrate the deferral threshold with conformal risk control, a method that picks a threshold from held-out data so that expected test loss provably stays below a target. The calibration is set up so that the counting loss becomes monotone in the threshold, which is the condition the guarantee requires.
Core Claim
A counting system that defers patches, rather than whole images, can meet a fixed error budget with far less human effort than whole-image abstention. The effort is measured as the number of heads a person must count in the deferred regions. The guarantee can be made valid by construction for a counting loss.
The construction works like this. An image is cut into patches $p$. The machine predicts $\hat c_p$ and a score $s_p$. Patches with $s_p \le \tau$ are counted by the machine; the rest go to a human, whom we treat as exact in the main analysis. The image-level loss is
$$L_i(\tau) = \min\Big(B,\ \frac{1}{\max(c_i,1)} \sum_{p:\, s_p \le \tau} |\hat c_p - c_p|\Big),$$
where $c_p$ is the true patch count, $c_i$ the true image count, and $B$ a cap that keeps the loss bounded. Two properties follow. First, $L_i$ upper-bounds the relative absolute error of the combined count. Second, because the sets of accepted patches are nested as $\tau$ grows, $L_i$ is non-decreasing in $\tau$. The raw count error $|\sum \hat c_p - c_i|$ does not have this property, since per-patch errors can cancel. That lack of monotonicity is exactly why Conformal Risk Control cannot be applied directly to mean absolute error.
The method solves this by bounding the error rather than tracking it exactly. On $n$ calibration images we pick the largest $\tau$ satisfying $\frac{n}{n+1}\hat R_n(\tau) + \frac{B}{n+1} \le \alpha$, where $\hat R_n$ is the mean of $L_i$. This gives $\mathbb{E}[L(\hat\tau)] \le \alpha$ on exchangeable test images. The guarantee is at the image level, so dependence between patches inside an image does not break it.
Relation to Prior Work
Two crowd-counting papers quantify uncertainty but never abstain, and they motivate this work. Crowd Counting with Decomposed Uncertainty (DUBNet) splits uncertainty into data noise and model ignorance, and recalibrates its intervals with isotonic regression; its 90% intervals contain the truth 89–91% of the time. Uncertainty Estimation and Sample Selection for Crowd Counting uses per-pixel variance only to choose training images for annotation. Neither paper has a test-time rule for handing work off, and neither has a detection branch. We reuse DUBNet-style variance as a competing score and ask whether hybrid disagreement beats it.
A second group of hybrid counters already makes internal selections, but always at 100% coverage. DecideNet blends detector and regressor outputs pixel by pixel through a learned attention map. Recurrent Attentive Zooming flags ambiguous regions and re-examines them at higher resolution. DAVE prunes outlier detections using spectral clustering. We keep their predictors and add the option they lack: sending a patch to a human when both branches are unreliable. We also attach a calibrated budget to that option, which DAVE's pruning heuristic does not have.
On the selective-prediction side, several papers supply the method or the evaluation. Selective Classification for Deep Neural Networks and Conformalized Selective Regression abstain on whole inputs; the second supplies our error–coverage evaluation and our whole-image baseline. Conformal Risk Control supplies the calibration step.
Two further papers constrain the design. Act or Defer shows that guarantees averaged over the whole population can hide error rates near 90% on rare subgroups. Crowd data has this structure: NWPU-Crowd spans 0 to 20,033 people per image. For that reason we also calibrate separately within density strata. Non-Exchangeable Conformal Risk Control provides the remedy we test when calibration and test scenes come from different datasets.
Research Program
The base predictor is a shared backbone with two heads, trained on NWPU-Crowd, whose box annotations make a detection head trainable. One head is a density regressor; the other is a point/box detector. The fused count is formed with DecideNet-style attention. Per patch we compute two quantities: the regressor's integral $\hat c^{reg}_p$ and the number of detections $\hat c^{det}_p$. The deferral score is their disagreement, scaled by expected count noise:
$$s_p = \frac{|\hat c^{reg}_p - \hat c^{det}_p|}{\sqrt{1+\hat c^{reg}_p}}.$$
The square-root scaling is chosen because Poisson-like count noise grows with the square root of the count. Without it, dense patches would be deferred for ordinary variability. We also train a variant in which a small head learns to predict $|\hat c_p - c_p|$ directly from both branches' features. This variant tests whether a learned score beats the hand-built one.
Calibration uses the loss above at several budgets $\alpha$. We calibrate in two ways. The first calibrates over all images together. The second calibrates stratum by stratum, grouping images by the machine's predicted total count so the rule can be used at test time without labels. The main outcome is human effort at matched guaranteed risk, reported as heads and as area in deferred patches. Secondary outcomes are realized risk and its variance per density stratum and on NWPU's 351 negative (empty) images.
We also address the tension between DUBNet-style variance and negative samples. DUBNet's variance loss lets a network down-weight ambiguous pixels. It could therefore learn high variance on foliage rather than learning to suppress it. We will check whether variance scores defer empty clutter patches less reliably than disagreement scores do.
Baselines
The naive reference point is do nothing: the fused hybrid counter at full coverage. It has zero human effort and an unbounded realized loss. For published full-coverage accuracy on these benchmarks, the file reports a validation MAE of 81.57 on NWPU-Crowd for SCAR (NWPU-Crowd). On UCF-QNRF it reports MAE 86 for CTN (Uncertainty Estimation and Sample Selection) and 105.6 for DUBNet. We report our base model's MAE against these figures only to show the predictor is competitive. Beating them is not the claim.
The strongest existing selective method is whole-image abstention with Conformalized Selective Regression (CSR). It fits conformal quantile intervals on the image count and abstains when the interval is too wide. It is the strongest competitor because it is the only cited method that controls error against coverage for continuous targets. On its own tabular benchmark it reports an error–coverage AUC of 0.328 on Communities, against 0.381 and 0.505 for the leading baselines. No reported selective-counting result was found on Lacuna for any crowd dataset. CSR will use the same backbone, calibration split, and tuning budget. We will translate its abstentions into human effort by counting every head in each rejected image. We count as beating CSR any result that reaches equal guaranteed risk with at least 30% fewer human-counted heads across $\alpha \in [0.02, 0.2]$ on NWPU-Crowd.
Within the patch framework we compare scores, all calibrated identically. The first is DUBNet-style variance, meaning bootstrap-head spread plus predicted noise. The second is MC-dropout variance. The third is predicted density alone, which simply defers the densest patches. The fourth is random deferral. Comparing these against hybrid disagreement is the ablation that isolates the proposal's key ingredient. A further ablation calibrates a density-only model with its own variance, which removes the detection branch entirely.
Why not just…?
Why not just abstain on whole images with CSR? This is simpler and already comes with guarantees. It is wasteful in a specific way: crowd scenes mix clear foreground, dense midground, and cluttered background. Rejecting the whole image makes a person recount thousands of easy heads to fix one bad region. The head-to-head effort curve in our experiments settles this. If image-level abstention matches patch deferral in effort, the patch machinery is not worth it and we will say so.
Why not just defer the densest patches? This is a real threat. Dense regions are where detectors collapse, as both DecideNet and DAVE observe, so density alone may catch most errors. Its blind spot is the opposite failure: sparse clutter where the regressor hallucinates, which NWPU-Crowd documents on negative images. Density-based deferral would accept those patches. We will report effort separately on negative and sparse images. If the density score ties hybrid disagreement everywhere, the disagreement score is unnecessary, and the contribution narrows to the monotone patch-loss calibration.
Why not just threshold detector confidence? Detector confidence does not degrade gracefully under congestion. DecideNet shows median detection scores falling with density. Detectors miss heads that span only a few pixels outright rather than scoring them low; RAZ_Loc reaches only 59.8% F1 on NWPU-Crowd. A patch with no boxes looks confidently empty. Detector confidence will be included as a score so the effort curves can show this.
Why not just use calibrated variance from DUBNet? Variance is the most principled competitor. Recalibration fixes the scale of its intervals, but it does not fix the ranking of patches, and ranking is what drives deferral. The concern that its variance loss hides clutter is testable on negative images. If variance matches disagreement, a density-only uncertainty model plus our calibration suffices, and we will report that outcome.
Proposed Experiments
The main experiment calibrates on half of the NWPU-Crowd validation set and tests on the other half, repeated over 50 random splits. For each score and each $\alpha$ we report realized mean loss, which must sit at or below $\alpha$ on average, and human effort. We also plot effort–risk curves together with CSR's error–coverage AUC and distance to the ideal point, recomputed on crowd data.
The first decisive check is conditional validity. Pooled calibration should show risk exceeding $\alpha$ in the highest-density stratum, as the Act-or-Defer result predicts. Per-stratum calibration should repair this at a measurable cost in effort.
The second is shift. We calibrate on NWPU-Crowd and test on UCF-QNRF and on ShanghaiTech Parts A and B. We measure how badly the guarantee breaks, and whether non-exchangeable reweighting restores it.
The third is robustness to imperfect human counts. We inject counting noise into deferred patches and track how far realized risk moves from the bound.
The fourth is transfer to generic objects. We repeat the main experiment on FSC147 with DAVE as the hybrid predictor, and test whether its verification-stage rejections, turned into a score, beat raw disagreement.
References
- Conformal Risk Control
- Crowd Counting with Decomposed Uncertainty
- Uncertainty Estimation and Sample Selection for Crowd Counting
- DecideNet: Counting Varying Density Crowds Through Attention Guided Detection and Density Estimation
- Recurrent Attentive Zooming for Joint Crowd Counting and Precise Localization
- DAVE - A Detect-and-Verify Paradigm for Low-Shot Counting
- Selective Classification for Deep Neural Networks
- Conformalized Selective Regression
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization
- Non-Exchangeable Conformal Risk Control