Zoom-Consistent Siamese Counting

Counting people in dense crowd photos usually requires an annotator to click a dot on every head, often thousands per image. Methods that avoid this rely on weaker signals: a few images labeled with only their total count, or "this image has more people than that one." These methods do reasonably well on sparse scenes and degrade on the densest ones. The reason is structural. A counter that sees a packed stadium at low resolution is far outside the regime where it is reliable, and nothing in its training ties its answer on the whole image to its answers on the parts. We propose a free supervisory signal that makes this tie. Cut an unlabeled image into disjoint tiles and enlarge each tile to full input size. The number of people in the whole image must exactly equal the sum of the counts in the enlarged tiles. A Siamese network is a pair of branches with identical weights, so both inputs are processed by literally the same model. We use one to look at the whole image and at its zoomed tiles, and train it so these two views agree.

In two sentences: the zoomed tiles are less crowded, so the model counts them more reliably, and forcing the whole-image prediction to match their sum pushes that reliability up into the dense regime without any dot labels. Unlike the ranking constraints used so far, this is an equality. It cannot be satisfied by predicting counts proportional to image area, because enlarging a tile multiplies its pixel area while leaving its true count unchanged.

Core Claim

Existing label-free crop constraints in crowd counting are inequalities. Leveraging Unlabeled Data for Crowd Counting by Learning to Rank (L2R) uses the fact that a sub-crop cannot contain more people than its parent and trains with a hinge loss on nested crops. S2FPR moves the same inequality into multi-depth feature maps. Glance to Count (GTC) trains a Siamese ranking network on human pairwise comparisons and then fits a linear map from its scores to counts using about 50 anchor images. These papers motivate our proposal, since they show that relational supervision from unlabeled images helps. They also mark its limit. An inequality says nothing about how much larger the parent's count is, and none of these methods relates predictions across resolution.

The closest mechanism is the inference-time partitioning in AFreeCA. It detects crowded regions, splits them, and counts the pieces at higher resolution, because its counter is reliable only at low density. That finding motivates our method. We turn the partitioning from a test-time workaround into a training signal, so the model itself becomes consistent across zoom levels.

Our claim is specific and testable. Training with a zoom-additive equality on unlabeled images, plus a small count-anchored set, should lower error on the densest images of ShanghaiTech Part A and UCF-QNRF. The improvement should hold for plain whole-image inference, not only when tiling is applied at test time. It should also exceed what the same budget achieves with ranking inequalities.

Research Program

The model has two parts. The first is a shared backbone (PVTv2, matching GTC). The second is a density head, which outputs a map whose pixel values sum to the count. Let $D(x)$ be the predicted density map for input $x$, and let $Z(I_k)$ be tile $I_k$ of image $I$ resized to the network's input resolution. Let $D(I)[R_k]$ be the sum of the full-image density map over the region $R_k$ that the tile occupies, and let $\Sigma D(\cdot)$ be the total of a map. The zoom-consistency loss compares counts in log space, so that very dense tiles do not dominate:

$$ L_{\text{zoom}} = \sum_{k} \Big| \log\big(1 + D(I)[R_k]\big) - \log\big(1 + \operatorname{sg}\,\Sigma D(Z(I_k))\big) \Big| $$

Here $\operatorname{sg}$ is an optional stop-gradient. When it is applied, the zoomed, less crowded view acts as a teacher for the coarse view.

The constraint is applied at the region level, so each image yields $k$ dense constraints rather than one global one. It also applies recursively. Tiles are themselves tiled for two or three levels, so a count from the most reliable, most zoomed regime can propagate up to the whole image.

Even a density-map model is not additive "for free" under this loss. Its per-pixel outputs are additive at a fixed resolution, but nothing makes them agree across rescaling, and rescaling is exactly where dense-scene errors arise.

The equality has one trivial solution: predicting zero everywhere. An anchor term breaks it. Given $N$ anchor images $a_j$ with known total counts $c_j$, the full objective is

$$ L = L_{\text{zoom}} + \lambda \sum_{j=1}^{N} \big| \log(1+\Sigma D(a_j)) - \log(1+c_j) \big|, $$

where $\lambda$ weights the anchor term. Because the anchor loss is applied during training rather than as GTC's post-hoc linear fit, gradients from calibration reshape the representation. This targets the non-linear distortion at high density that Weakly-Supervised Crowd Counting Learns from Sorting Rather Than Locations reports on Part A.

We study three supervision regimes. In the first, the model gets $N=50$ count-anchored images plus the rest of the training set unlabeled, which matches GTC's anchor budget. In the second, every training image has a count but no dots, the regime of Towards Using Count-level Weak Supervision for Crowd Counting (MATT). In the third, there are no human labels at all, and the encoder is initialized from SYRAC-style diffusion ranking pretraining with its noisy synthetic anchoring. Beyond accuracy, we will measure held-out zoom-consistency error as a diagnostic and analyze per-density-bin error to locate where the gains occur.

Baselines

The do-nothing reference is the same backbone and density head trained only on the anchor loss, with no unlabeled-data term. It shows how much the 50 anchors buy by themselves. In the count-level regime, the corresponding reference is naive count supervision, for which MATT reports MAE 89.3 on ShanghaiTech Part A. MATT's multi-scale auxiliary branches bring this to 80.1, so MATT is the main competitor in that regime.

The strongest existing method under our anchor budget is GTC. It reports MAE 53.4 on ShanghaiTech Part A and 92.3 on UCF-QNRF, using human pairwise comparisons expanded by transitivity plus a 50-image linear anchor fit. We will rerun it with its released configuration on our splits and backbone. We will report our method both without pairwise comparisons, which is strictly less human input than GTC, and with GTC's comparisons added as an extra ranking loss. Beating GTC means lower MAE on Part A and QNRF with no more human input, and specifically lower MAE in the top density quintile.

For semi-supervised context, we include two points of comparison. The first is the Gaussian-process pseudo-labeling of Learning to Count in the Crowd from Limited Labeled Data, which reports Part A MAE falling from 118 to 102 with 5% dot labels. The second is S2FPR, which reports MAE 78.4 on Part A with 50% labels and 251.52 on UCF_CC_50, against L2R's 279.6. If we match these using counts on only 50 images, that is a strong annotation-cost result.

For the label-free regime, the references are SYRAC, with ShanghaiTech B MAE 49.0 and JHU-CROWD++ MAE 194.0, and Completely Self-supervised Crowd Counting via Distribution Matching, with Part A MAE 197.3. For AFreeCA we found no reported ShanghaiTech B or JHU result on Lacuna, so we will rerun it.

The key ablation replaces our equality with L2R's hinge inequality on the same zoomed tile pairs. Data, schedule, and tuning budget stay identical, with 20 trials of hyperparameter search per method on a held-out validation split taken from training.

Why not just…?

Why not just tile at test time, as AFreeCA does? This is the real threat. If GTC or the anchor-only baseline, run with recursive test-time tiling, matches our model, then the gain comes from inference and the training loss adds nothing. We will therefore report every method both with and without test-time tiling. The idea survives only if two things hold. First, our model is better under plain whole-image inference. Second, it remains better when both methods are tiled, which would show that the tile-level counter itself improved through the shared weights. If neither holds, we will report that consistency training is unnecessary given tiled inference.

Why not just use the ranking trick from L2R? The hinge ablation answers this directly. Our argument is that an inequality is satisfied by any monotone distortion of the true count, while the equality rules most of those out. The comparison holds the data and the crop pairs fixed and changes only the form of the constraint. If the hinge matches the equality on high-density bins, our central claim is wrong.

Why not just regress counts with a DINO-initialized transformer? Learning to Count Anything shows that this works for class-agnostic counting, with MAE 17.12 on FSC-147. However, sorting-based work found that pure count regression on WorldExpo'10 gave MAE 20.1 versus 9.6 once relational constraints were added. MATT similarly found that naive count supervision produces arbitrary blobs. We include this regressor, trained on the anchor set and in the count-level regime, as a baseline.

Why not just pseudo-label with uncertainty filtering? Gaussian-process pseudo-labeling needs labeled dot maps to interpolate from, and our anchor regime has none. Pseudo-labeling also does not address the cross-scale mismatch that drives dense-scene error. As a fair test, we will combine count-only anchors with confidence-filtered self-training at the same budget.

Proposed Experiments

The main evaluation covers ShanghaiTech Part A and B, UCF-QNRF, UCF_CC_50 (five-fold), and JHU-CROWD++, reporting MAE and RMSE. As a transfer check beyond crowds, we also evaluate on CARPK, where AFreeCA reports MAE 9.35. Every table is broken out by ground-truth density quintile.

The decisive checks are four. First, whole-image MAE in the top density quintile of Part A and QNRF should fall relative to GTC and to the hinge ablation. Second, the advantage should persist when all methods use test-time tiling. Third, recursion depth should matter: one level of tiling should be measurably worse than two or three on the densest images. Fourth, held-out zoom-consistency error should correlate with counting error across checkpoints, which would show that the signal is tracking what we claim.

The no-label regime is a secondary test of whether zoom consistency on top of SYRAC pretraining improves on SYRAC's 194.0 on JHU-CROWD++. A null result there would not affect the main claim, which concerns the 50-anchor and count-level regimes.