Fine-grained Domain Adaptive Crowd Counting via Point-derived Segmentation

Yongtuo Liu, Dan Xu, Sucheng Ren, Hanjie Wu, Hongmin Cai, Shengfeng He

TL;DR

Researchers have developed a new framework called CACC (Crowd-aware domain Adaptation for Crowd Counting) that significantly improves how AI counts people in unfamiliar environments. By "disentangling" the crowd from the background using a clever weakly-supervised segmentation trick, the model focuses on human features while ignoring distracting environmental changes like buildings, trees, or weather.

What Problem Does This Paper Address?

Crowd counting models are notoriously sensitive to domain shift. If you train a model on sunny, synthetic images (Source Domain) and deploy it in a rainy, real-world city square (Target Domain), its accuracy usually plummets.

The core challenge is that existing "Unsupervised Domain Adaptation" (UDA) methods try to align the entire image at once. However, backgrounds vary wildly between locations—one scene might be a paved plaza while another is a park with dense foliage. Forcing a model to align these disparate backgrounds actually confuses it, making it harder to distinguish between a person and a tree. Furthermore, different locations have different "crowd signatures"; a stadium has a very different density distribution than a sidewalk, yet most models try to force the target domain to look exactly like the source domain's density.

Methods

The authors propose the Crowd-aware domain Adaptation framework for Crowd Counting (CACC), which consists of three main innovations:

  1. Point-derived Crowd Segmentation (PCS): Standard crowd datasets only provide "dot" annotations on heads, not full outlines of people. To separate people from the background without extra labeling work, the authors used Multiple Instance Learning (MIL). They treat image patches as "bags" of pixels; if a patch has a dot, it's a "crowd bag." This allows the model to learn a rough segmentation mask (where the people are) in a weakly-supervised way.
  2. Crowd Region Transfer (CRT): Instead of aligning the whole image, this module uses the masks from the PCS to focus adversarial training specifically on the crowd regions. It discards domain-specific background features and forces the model to learn "domain-invariant" features of human heads.
  3. Crowd Density Alignment (CDA): Rather than using the source domain's density distribution as a template, this module generates "pseudo-labels" based on the target domain's own predicted distribution. This respects the fact that a subway station in Tokyo has a naturally different crowd density than a street in London.

Datasets

The study utilized six major benchmarks to test various "real-world" shift scenarios:

  • Synthetic-to-Real: GCC (Source) → ShanghaiTech Part A & B (Target).
  • Fixed-to-Fickle: ShanghaiTech Part B (fixed camera) → ShanghaiTech Part A (varied scenes).
  • Weather Changes: ShanghaiTech Part A (Normal) → JHU-CROWD (Bad weather).
  • Other Benchmarks: MALL and UCSD datasets were used for cross-dataset comparisons.

Key Insights and Ideas

  • Background Alignment is a Distraction: The most counter-intuitive finding is that trying to make the backgrounds match across domains actually harms performance. By explicitly masking out the background during feature alignment, the model becomes much more robust.
  • Heads are the "Anchor": Because crowd counting is "head-centric," the weakly-supervised segmentation naturally highlights heads. This "head-highlighted" feature alignment is more effective than general body alignment.
  • Self-Regularization over Source-Template: Forcing a target domain to match a source domain's density distribution is a "square peg in a round hole" problem. Letting the target domain regularize itself via pseudo-labels derived from its own segmentation leads to much more accurate local density maps.

Key Results

The CACC method consistently outperformed previous state-of-the-art (SOTA) methods across all tested scenarios:

  • Synthetic-to-Real: On the GCC to ShanghaiTech Part B task, CACC achieved a Mean Absolute Error (MAE) of 13.5, beating the previous best unsupervised method (ASNet) which sat at 14.6.
  • Performance Gains: The model showed relative gains in accuracy (MAE/RMSE) of up to 35.4%/26.8% on the challenging ShanghaiTech Part A dataset compared to models without adaptation.
  • Qualitative Improvement: Visual density maps show that CACC is significantly better at suppressing "false positives" in the background (like mistaking a bush for a person) compared to holistic alignment methods.

Limitations and Future Work

While the paper demonstrates a major leap in unsupervised adaptation, some limitations remain:

  • Initial Segmentation Quality: The entire pipeline relies on the quality of the initial "point-derived" segmentation. In extremely noisy or low-resolution images, if the initial MIL-based segmentation fails, the subsequent alignment modules (CRT and CDA) will likely struggle.
  • Computational Overhead: The framework involves multiple sub-networks (segmenter, counter, and discriminators), which might increase training time compared to simpler holistic methods.
  • Future Directions: The authors suggest that exploring more sophisticated ways to handle the "noisy" nature of the weakly-supervised masks could further refine the feature alignment process.

Summary updated

Abstract

Due to domain shift, a large performance drop is usually observed when a trained crowd counting model is deployed in the wild. While existing domain-adaptive crowd counting methods achieve promising results, they typically regard each crowd image as a whole and reduce domain discrepancies in a holistic manner, thus limiting further improvement of domain adaptation performance. To this end, we propose to untangle domain-invariant crowd and domain-specific background from crowd images and design a fine-grained domain adaption method for crowd counting. Specifically, to disentangle crowd from background, we propose to learn crowd segmentation from point-level crowd counting annotations in a weakly-supervised manner. Based on the derived segmentation, we design a crowd-aware domain adaptation mechanism consisting of two crowd-aware adaptation modules, i.e., Crowd Region Transfer (CRT) and Crowd Density Alignment (CDA). The CRT module is designed to guide crowd features transfer across domains beyond background distractions. The CDA module dedicates to regularising target-domain crowd density generation by its own crowd density distribution. Our method outperforms previous approaches consistently in the widely-used adaptation scenarios.
BibTeX
@inproceedings{liu2023fine,
  author = {Yongtuo Liu and Dan Xu and Sucheng Ren and Hanjie Wu and Hongmin Cai and Shengfeng He},
  title = {{Fine-grained Domain Adaptive Crowd Counting via Point-derived Segmentation}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00403},
}

Loading related content...

Figure