TL;DR
SYRAC (Synthesize, Rank, and Count) eliminates human annotation for crowd counting by leveraging latent diffusion models in a two-stage training scheme. By combining reliable pairwise ranking data (derived from diffusion-based pedestrian removal) with a linear probe trained on noisy count-prompted synthetic images, SYRAC sets a new state of the art in truly unsupervised crowd counting.
What Problem Does This Paper Address?
Standard supervised crowd counting relies on manual point annotations to generate density maps—an excruciatingly tedious task that can take over 5 seconds per pedestrian (amounting to thousands of hours for large benchmarks).
While unsupervised crowd counting eliminates this burden, existing approaches are either constrained by rigid theoretical assumptions (such as assuming crowd numbers follow an exact power law) or subtly rely on labeled test sets for early stopping. Generating training data with text-to-image models like Stable Diffusion seems like an obvious workaround, but text-to-image models struggle with numerical reasoning: prompting for "a group of 20 people" often yields anywhere from 10 to 44 individuals, introducing severe label noise that causes neural networks to overfit and fail.
Methods
SYRAC sidesteps the counting inaccuracy of diffusion models by decomposing data generation and training into two complementary phases:
- Synthetic Ranking Pre-training (Weak, Reliable Signal):
- Instead of asking Stable Diffusion (SD 2.1) to count, the authors use image-to-image synthesis to remove pedestrians from real, unlabeled images using negative prompts (e.g., "people, crowds, pedestrians").
- This produces synthetic images with fewer pedestrians while retaining original scene perspectives and background textures.
- A Siamese network (ResNet-50 backbone) is pre-trained on these pairs using a pairwise ranking loss ($c^{\text{real}} \ge c^{\text{syn}}$). The ranking decoder applies a non-negativity constraint on its weights, preventing the model from using background artifacts to artificially deflate counts.
- Noisy Synthetic Linear Probing (Strong, Noisy Signal):
- The authors synthesize a small dataset (420 images total) using text-to-image prompts specifying exact counts: $N \in \{0, 1, 5, 10, 50, 100, 200\}$.
- To prevent the network from overfitting to the label noise inherent in these synthetic prompts, the pre-trained ResNet-50 feature extractor is frozen.
- A single linear layer is trained on top of the frozen representations via Mean Squared Error (MSE), calibrating the relative ranking features into absolute numerical counts.
- Inference: Because the synthetic calibration is capped at 200 people, test images are partitioned into a grid of patches (e.g., $2 \times 2$) to keep local crowd counts within the calibrated range before summing.
+-----------------------------------------------------------+
| 1. Image-to-Image Pedestrian Removal |
| Real Image ──> [Stable Diffusion + Neg Prompt] ──> |
| Image Pair (Real >= Synthetic) |
| │ |
| ▼ |
| Train Siamese ResNet-50 (Pairwise Ranking Loss) |
+-----------------------------------------------------------+
│
(Freeze Weights)
│
+-------------------------▼---------------------------------+
| 2. Text-to-Image Calibration |
| "A group of {N} people" ──> Synthetic Images |
| │ |
| ▼ |
| Fit Linear Layer on Frozen Features (MSE Loss) |
+-----------------------------------------------------------+
Datasets
- Evaluation Benchmarks: Evaluated across four standard datasets:
- ShanghaiTech Part A (SHA) (482 images, dense crowds)
- ShanghaiTech Part B (SHB) (716 images, sparse crowds)
- UCF-QNRF (1,535 images, extremely dense crowds)
- JHU-Crowd++ (4,372 images, diverse environmental conditions)
- Synthetic Datasets Introduced:
- $D^{\text{syn}}_{\text{rank}}$: 4 synthetic pedestrian-reduced images generated per real source image.
- $D^{\text{syn}}_{\text{count}}$: 420 synthetic images generated across 7 discrete count prompts (60 images per count bucket).
Key Insights and Ideas
- Relative changes beat absolute synthesis: Diffusion models are poor counters, but they are remarkably consistent at object removal when guided by negative prompts. Translating an intractable absolute generation task into a relative ranking task yields a reliable supervisory signal.
- Linear probing as noise defense: Training end-to-end on synthetic count prompts collapses due to severe label noise. Freezing the ranking features constrains the hypothesis space strictly to crowd-relevant representations, rendering linear probing robust against noisy labels.
- Truly unsupervised evaluation: The authors identify that prior state-of-the-art methods (like CrowdCLIP) used labeled ground-truth test data for early stopping. SYRAC performs model selection using a 15% validation split of its unlabeled synthetic ranking data, preserving strict unsupervised constraints.
Key Results
SYRAC achieves state-of-the-art results among unsupervised baselines across multiple benchmarks (measured in Mean Absolute Error, MAE, and Mean Squared Error, MSE):
- ShanghaiTech B: Achieves 49.0 MAE / 60.3 MSE, outperforming prior SOTA CrowdCLIP (69.3 MAE) and CSS-CCNN.
- JHU-Crowd++: Achieves 194.0 MAE / 583.9 MSE, beating CrowdCLIP (213.7 MAE) and CSS-CCNN (217.6 MAE).
- ShanghaiTech A & UCF-QNRF: Scores 196.0 MAE on SHA and 390.0 MAE on QNRF, consistently outperforming the fully unsupervised CSS-CCNN (197.3 and 437.0 MAE, respectively).
- Ablation on Pre-training: Training on the synthetic count data without ranking pre-training results in disastrous performance (e.g., SHB MAE jumps from 49.0 to 113.6), proving the ranking stage is vital for mitigating label noise.
Limitations and Future Work
- High-Density Ceilings: Prompt-based synthesis degrades rapidly above 200 people, necessitating patch division at test time.
- Fixed Grid Sensitivity: Performance varies with grid size; sparse datasets (SHB) perform best with full images ($1 \times 1$), whereas denser scenes (UCF-QNRF) favor finer grids ($4 \times 4$). An adaptive density-guided cropping mechanism could improve results.
- Manual Prompt Engineering: Prompts were hand-crafted via heuristic trial-and-error; automated prompt tuning could further improve synthetic fidelity.
Summary updated