Counting people in crowds or cells under a microscope is usually done by predicting a "density map": an image-sized heat map whose pixel values sum to the number of objects. Such counters break when moved to a new domain, for example from synthetic video-game crowds to real photographs. They also break worst on the smallest objects, because every pooling step in the network blurs an 8×8-pixel head into the background before the prediction head ever sees it. The standard fix for domain shift is adversarial alignment. A second network, the "discriminator", tries to tell which domain a feature came from, and the counter is trained to fool it. That fix is usually applied at the network's lowest-resolution layer, which is exactly where tiny objects have already been erased. Alignment cannot restore signal that pooling has already thrown away.
We propose a target-side training signal that only density regression offers. Density is additive. If the network reads out a small density map at every encoder stage, the mass inside each coarse cell should equal the mass its finer child cells contained one stage earlier. On unlabeled target images, any violation of this rule shows where the network lost objects during downsampling. Penalizing those violations trains the pooling layers to keep target-domain tiny objects, and it does so without any target labels. Adversarial alignment stays in the method, but it plays a supporting role: it suppresses background false positives at the output, while conservation protects small objects inside the encoder.
Core Claim
Our claim is that cross-domain counting error is concentrated at the small-object end of the size distribution, and that most of this error enters through downsampling rather than through feature-style mismatch. A local mass-conservation loss between per-stage density readouts, applied to target images and combined with content-aware pooling, should reduce small-object error more than any form of feature alignment. Feature alignment here means alignment at the bottleneck, at every skip connection, or at the output.
The loss is
$$L_{\text{cons}} = \sum_{s=2}^{S} \sum_{c \in \mathcal{C}_s} \Big| \sum_{x \in \text{ch}(c)} D_{s-1}(x) - D_s(c) \Big|,$$
with the following symbols. $D_s$ is the density predicted by a light head attached to encoder stage $s$ (of $S$ stages). $\mathcal{C}_s$ is the set of spatial cells at stage $s$. $\text{ch}(c)$ is the set of finer-stage cells that pool into cell $c$. On source images, each $D_s$ is also supervised with the ground-truth density sum-pooled to that resolution, so all readouts are anchored to true counts. On target images, only $L_{\text{cons}}$ and the adversarial term apply.
The gradient of this loss flows into the pooling weights. We therefore replace fixed pooling with the learnable importance-weighted pooling of LIP: Local Importance-Based Pooling. LIP predicts a per-pixel importance map that reweights features before downsampling, and it raised COCO small-object AP from 21.9% to 24.0%. LIP was only ever trained and tested within a single domain. Here its importance module receives a label-free target signal that tells it which target pixels it is dropping.
Positioning Against Prior Work
Several papers motivate the gap. Automatic microscopic cell counting by use of unsupervised adversarial domain adaptation and supervised density regression aligns encoder bottleneck features using a critic that estimates the Wasserstein distance between domains, then reuses the source decoder. FAN-Net attaches a gradient-reversal domain classifier to a U-Net for lesion segmentation and normalizes the low-frequency Fourier amplitude of features. Both align at a single place and treat downsampling as fixed. Neither measures where the lost signal goes.
Two crowd-counting methods supply components and set the bar. ASNet applies global and patch discriminators to predicted density maps and reweights the source loss by transferability scores. We reuse its output-space patch discriminator as our background-suppression term. DACC and SE Cycle GAN from Learning From Synthetic Data for Crowd Counting in the Wild work at the pixel level instead: they translate images between domains with structure-preserving losses, and DACC adds pseudo-labels. Our method keeps the counter's training single-stage and works entirely on features.
One paper constrains the design. Self-Supervised Domain Adaptation in Crowd Counting abandoned density maps for discrete point proposals, arguing that density maps are too fuzzy to give a crisp target-side signal. Conservation is a direct answer to that argument, because the additive rule is exact even when the map is blurry. We include that method as a baseline to see whether the argument survives the comparison.
Composition Loss supplies the idea of supervising densities at several resolutions; its hierarchical loss cut UCF-QNRF MAE from 163 to 132. It does this only with labels and only on source data. We turn the same multi-resolution structure into an unsupervised constraint on target images.
Research Program
We build a density regressor with a VGG- or ResNet-style encoder, LIP pooling, a density head at each stage, and a U-Net decoder with skip connections. The training objective has three parts: supervised multi-scale density loss on source images, $L_{\text{cons}}$ on target images, and an ASNet-style patch discriminator on the final density map.
One design question needs a principled answer: which side of each conservation pair receives the gradient. A finer stage sees small objects but has less context, so it is more prone to false positives on foliage or facades. We will compare three choices. The first is symmetric gradients. The second detaches the fine side, so that only the coarse side and the pooling weights are pushed. The third is a one-sided hinge that only penalizes coarse mass falling below fine mass. The hypothesis behind the third choice is that pooling loses mass rather than inventing it. The adversarial output term is meant to stop the fine heads from inflating counts on background.
The main analysis tool is a size-stratified error. We estimate each annotated head's size from the distance to its k nearest neighbours, following the geometry-adaptive kernel logic of MCNN. We then bin heads by size and compute count error within the matching density-map regions. If the core claim holds, the gap between source-only and adapted models should be largest in the smallest bins, and conservation should close most of it.
We also run a controlled degradation protocol: target images are downsampled by factors of 2 and 4 before inference. This turns "resolution degradation" from an informal notion into an axis we can plot error against.
Baselines
Source-only reference. The naive point of comparison is the same architecture trained only on source data, with fixed pooling. The file reports this gap for GCC→ShanghaiTech Part A as MAE 206.7 without adaptation versus 112.4 for DACC, in DACC. For synthetic bacteria→hESC cells, it reports 170.06 versus 35.82 in Automatic microscopic cell counting. We retrain our own source-only model so that every comparison shares a backbone.
Strongest crowd adaptation methods. These are the pixel-translation methods, DACC and SE Cycle GAN, and the feature-scoring method ASNet. Their reported numbers are DACC MAE 112.4 on GCC→SHA, SE Cycle GAN MAE 19.9 on GCC→SHB, and ASNet MAE 13.59 on SHA→SHB, close to the 13.6 of a fully supervised counter. The point-proposal method reports MAE 69.21 on SHB→SHA. We will quote these numbers, but protocols and backbones differ, so we will also reimplement ASNet and the point-proposal method on our backbone. Each gets the same data, the same number of training iterations, and an equal hyperparameter search (the same number of configurations tuned on a held-out labeled source split, with no target labels used for selection).
For GCC→UCF-QNRF, the file reports SFCN at MAE 102.0, but the summary does not make clear whether that model was fine-tuned on real labels. We will therefore not treat it as an unsupervised target. For DACC on UCF-QNRF there is no reported result on Lacuna.
Beating the strongest baseline means three things. Our method must lower overall MAE on GCC→SHA below 112.4 and on GCC→SHB below 19.9. It must also beat our reimplemented ASNet under the matched protocol. Finally, it must show a larger gain in the smallest size bins than in the largest.
Ablations isolating the key ingredient. The first ablation removes $L_{\text{cons}}$ while keeping LIP and the adversarial head, which isolates conservation. The second keeps conservation but uses fixed average pooling, which tests whether a learnable downsampling decision is needed. The third replaces conservation with a global-only count match across stages, which tests whether locality matters.
Why not just…?
Why not just align features at every skip connection with a discriminator? Multi-level adversarial alignment does address the fact that skip connections carry unadapted target style into the decoder. It cannot, however, restore objects that pooling removed before the skip was formed. The aligned features would be domain-invariant and still missing the tiny heads. We run this as a baseline with one discriminator per stage. The decisive check is the size-stratified error: if per-skip alignment matches our method in the smallest bins, our core claim is wrong.
Why not just renormalize the skip features? The cheap option is to re-estimate target statistics in the normalization layers, or to normalize low-frequency amplitude as FAN-Net does. This is a genuine threat for style shifts such as microscopy staining, where FAN-Net's normalization drove the Dice gain from 0.4712 to 0.5591. We expect it to help large and mid-sized objects and to leave small-object loss untouched, because normalization does not change what pooling keeps. Both variants are run, alone and combined with conservation. If they close the small-object gap by themselves, we will report that the problem is style rather than resolution.
Why not just upsample the target images at test time? Enlarging the input makes heads bigger than the pooling footprint and directly attacks resolution degradation. This is plausibly as effective as our method and is the most serious alternative. We run it at 1.5× and 2× and report error together with compute. We also run a related training-time variant that enforces count consistency between two input scales. Our method has to win either on accuracy at matched compute or on accuracy under the 2× and 4× degradation protocol, where upsampling cannot recover pixels that were never captured. If it wins on neither, the internal conservation idea offers nothing beyond input rescaling.
Why not just annotate a handful of target images? In the cell setting, five labeled real images gave MAE 39.84 versus 35.82 for unsupervised adaptation (Automatic microscopic cell counting). Few-shot labeling is therefore competitive but not dominant. We include a 5-image fine-tuning baseline on each target and test whether conservation still helps when added on top of it.
Proposed Experiments
We test on GCC→ShanghaiTech A, GCC→ShanghaiTech B and GCC→UCF-QNRF; on ShanghaiTech A→B and B→A; on ShanghaiTech→TRANCOS, where ASNet reports error dropping from 13.78 to 4.77; and on synthetic bacteria→hESC microscopy. We report MAE and MSE, size-stratified MAE, the localization AUC from Composition Loss on UCF-QNRF, and background false-positive mass (the predicted density on annotated-empty regions).
Three results would settle the question. First, in the source-only model, the stage-to-stage conservation violation on target images should correlate with per-image count error. If it does not, the loss is a bad proxy. Second, conservation should cut small-bin error substantially more than per-skip alignment, skip renormalization, or test-time upsampling at equal compute. Third, the gain from conservation should grow as targets are degraded by 2× and 4×, since that is the regime the method is built for. A flat curve would mean we have rediscovered generic regularization. We will also report background false-positive mass for every gradient variant, because conservation that inflates counts on foliage would fail the purpose even if small-object recall improved.
References
- LIP: Local Importance-Based Pooling
- Automatic microscopic cell counting by use of unsupervised adversarial domain adaptation and supervised density regression
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- Coarse to Fine: Domain Adaptive Crowd Counting via Adversarial Scoring Network
- Domain-Adaptive Crowd Counting via High-Quality Image Translation and Density Reconstruction
- Learning From Synthetic Data for Crowd Counting in the Wild
- Self-supervised Domain Adaptation in Crowd Counting
- Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds
- Single-Image Crowd Counting via Multi-Column Convolutional Neural Network