SYRAC: Synthesize, Rank, and Count

Adriano C. D'Alessandro, Ali Mahdavi-Amiri, Ghassan Hamarneh

Abstract

Crowd counting is a critical task in computer vision, with several important applications. However, existing counting methods rely on labor-intensive density map annotations, necessitating the manual localization of each individual pedestrian. While recent efforts have attempted to alleviate the annotation burden through weakly or semi-supervised learning, these approaches fall short of significantly reducing the workload. We propose a novel approach to eliminate the annotation burden by leveraging latent diffusion models to generate synthetic data. However, these models struggle to reliably understand object quantities, leading to noisy annotations when prompted to produce images with a specific quantity of objects. To address this, we use latent diffusion models to create two types of synthetic data: one by removing pedestrians from real images, which generates ranked image pairs with a weak but reliable object quantity signal, and the other by generating synthetic images with a predetermined number of objects, offering a strong but noisy counting signal. Our method utilizes the ranking image pairs for pre-training and then fits a linear layer to the noisy synthetic images using these crowd quantity features. We report state-of-the-art results for unsupervised crowd counting.

Introduction

Crowd counting—estimating the number of people in an image—is vital for urban planning, public safety, and event management. Despite its importance, it remains a "data-hungry" task. Traditional supervised models require human annotators to manually click on every individual in a photo to create a density map. In dense scenes, this process can take over 5 seconds per person, meaning a large dataset can require thousands of hours of manual labor.

While semi-supervised and weakly-supervised methods have attempted to reduce this burden, "unsupervised" counting (training without any human-provided labels) remains the ultimate challenge. The core difficulty lies in teaching a model to recognize "quantity" without ever providing it with the true count of a real image. SYRAC (Synthesize, Rank, and Count) addresses this by leveraging the generative power of latent diffusion models to create a self-contained training environment.

Methodology

The SYRAC framework utilizes Stable Diffusion (SD) to generate two distinct types of synthetic training signals, bypassing the need for human annotation entirely.

1. Synthetic Data Generation

To overcome the limitations of current generative models in understanding exact quantities, the authors propose a dual-signal approach:

  • Ranking Pairs (The "Weak" Signal): The system takes real, unlabeled images and uses image-to-image diffusion to remove pedestrians. This creates a pair where the original image is guaranteed to have more people than the synthetic version. This provides a highly reliable relative signal for the model to learn what "more vs. less" looks like.
  • Noisy Counts (The "Strong" Signal): The system prompts Stable Diffusion to generate images with specific numbers (e.g., "A group of 50 people"). While generative models are currently imprecise at exact counting, these images provide a rough absolute scale for the model to learn numerical values.
The SYRAC pipeline: (a) generating reliable ranking pairs by removing pedestrians from real images, and (b) generating noisy synthetic images with specific count prompts.

2. Two-Step Learning Process

The model architecture, typically a ResNet50-based Siamese network, is trained in two stages:

  • Pre-training: The network is trained on the ranking pairs. It learns to extract features that specifically correlate with crowd density and relative quantity.
  • Linear Probing: The feature encoder is frozen, and a simple linear layer is trained using the noisy synthetic images. By freezing the encoder, the model is forced to use the "quantity features" it already learned, preventing it from overfitting to the inaccuracies (noise) in the synthetic labels.

Experiments and Results

SYRAC was evaluated on four major crowd-counting benchmarks: ShanghaiTech Part A (SHA), ShanghaiTech Part B (SHB), UCF-QNRF, and JHU-Crowd++. Crucially, zero human labels from these datasets were used during the training process.

The model achieved State-of-the-Art (SOTA) results for unsupervised crowd counting across all benchmarks:

  • On ShanghaiTech B, it achieved a Mean Absolute Error (MAE) of 49.0, significantly outperforming the previous best unsupervised method (CrowdCLIP at 69.3).
  • On JHU-Crowd++, it achieved an MAE of 194.0, surpassing the previous SOTA (CSS-CCNN at 217.6).

Qualitative analysis using UMAP visualizations showed that the model's features formed a clear, continuous ordering based on actual crowd density, proving it learned the underlying concept of quantity rather than just memorizing textures.

Discussion and Key Insights

The success of SYRAC stems from several key conceptual shifts:

  • Diffusion as a Data Engine: Rather than just generating pretty pictures, generative models can be used as automated manipulation tools to create perfect, relative supervision signals (e.g., subtraction).
  • Balancing Reliability and Strength: By combining reliable ranking with noisy absolute counts, the model gains both the "what" (features of quantity) and the "how much" (numerical scale).
  • Architectural Robustness: Freezing the encoder during linear probing is a critical design choice that allows the model to benefit from noisy data without being misled by its errors.

Limitations and Future Work

Despite its success, SYRAC faces a "counting gap" where Stable Diffusion's accuracy degrades as the requested number of people increases. Currently, synthetic prompts are limited to 200 people to avoid extreme noise, which may impact performance in ultra-dense "mega-crowd" scenarios. Additionally, a domain gap remains between synthetic and real pedestrians. Future work will likely focus on improving the spatial and numerical understanding of generative models to further close the gap between unsupervised and fully supervised performance.

Summary updated

Abstract

Crowd counting is a critical task in computer vision, with several important applications. However, existing counting methods rely on labor-intensive density map annotations, necessitating the manual localization of each individual pedestrian. While recent efforts have attempted to alleviate the annotation burden through weakly or semi-supervised learning, these approaches fall short of significantly reducing the workload. We propose a novel approach to eliminate the annotation burden by leveraging latent diffusion models to generate synthetic data. However, these models struggle to reliably understand object quantities, leading to noisy annotations when prompted to produce images with a specific quantity of objects. To address this, we use latent diffusion models to create two types of synthetic data: one by removing pedestrians from real images, which generates ranked image pairs with a weak but reliable object quantity signal, and the other by generating synthetic images with a predetermined number of objects, offering a strong but noisy counting signal. Our method utilizes the ranking image pairs for pre-training and then fits a linear layer to the noisy synthetic images using these crowd quantity features. We report state-of-the-art results for unsupervised crowd counting.
BibTeX
@misc{dalessandro2023syrac,
  author = {Adriano C. D'Alessandro and Ali Mahdavi-Amiri and Ghassan Hamarneh},
  title = {{SYRAC: Synthesize, Rank, and Count}},
  year = {2023},
  eprint = {2310.01662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01662},
}
Figure