Optimal Transport Minimization: Crowd Localization on Density Maps for Semi-Supervised Counting

Wei Lin, Antoni B. Chan

Introduction

Modern crowd-counting systems primarily rely on deep learning to generate "density maps"—blurry representations where pixel values indicate the local concentration of people. While effective for estimating total counts, these maps often fail at precise localization, making it difficult to pinpoint the exact coordinates of individuals in dense or noisy environments. Furthermore, the high cost of manually labeling thousands of heads in images limits the scalability of supervised models.

To address these challenges, researchers have introduced Optimal Transport Minimization (OT-M), a parameter-free algorithm that converts blurry density maps into precise point locations. By integrating OT-M into a semi-supervised learning (SSL) framework, the system can leverage small amounts of labeled data alongside large unlabeled datasets to achieve state-of-the-art performance in both counting and localization.

The proposed pipeline: A CNN generates a density map, which OT-M then converts into a precise point map for localization.

The OT-M Algorithm

The core innovation is the OT-M post-processing step, which requires no additional training. It treats a density map as a distribution of "mass" and seeks to move that mass to a set of discrete points (representing people) while minimizing the total "transport cost."

The algorithm operates through an iterative two-step process:

  1. OT-step: It calculates an optimal transport plan (using the Sinkhorn algorithm) to map the continuous density to current discrete point estimates.
  2. M-step: It moves the points to the "barycenter" (the center of mass) of the density assigned to them in the previous step.
The iterative OT-M process: (a) initial density map, (b-c) points moving toward centers of mass over successive iterations to achieve precise localization.

Semi-Supervised Learning with Hard Pseudo-Labels

Most semi-supervised counting methods use "soft" pseudo-labels (blurry density maps), which provide a weaker training signal than the "hard" labels (exact points) used in other computer vision tasks. By using OT-M, this framework generates high-quality hard pseudo-labels from unlabeled data.

To manage the noise inherent in these pseudo-labels, the authors developed the Confidence-weighted Generalized Loss (C-GL). Implemented within a "Mean-Teacher" setup, C-GL includes a gating mechanism to filter out contradictory gradients and a weighting strategy that prioritizes unlabeled data where the teacher and student models show high consistency.

Key Results and Insights

The researchers evaluated the framework on major benchmarks, including UCF-QNRF, NWPU-Crowd, and ShanghaiTech. Key findings include:

  • Superior Localization: On the UCF-QNRF dataset, OT-M achieved an F-measure of 0.912, significantly outperforming traditional methods like Local Maximum (0.807).
  • Efficiency in Data-Scarce Scenarios: In semi-supervised tests with only 5% labeled data, the method reduced the Mean Absolute Error (MAE) on the ShanghaiTech Part A dataset from 92.9 to 86.0, setting a new state-of-the-art for SSL counting.
  • Plug-and-Play Flexibility: Since OT-M is parameter-free, it can be integrated into any existing density-map-based counting model to provide immediate localization capabilities.

Conclusion

Optimal Transport Minimization bridges the gap between density estimation and object localization. By converting "soft" predictions into "hard" point maps, it provides a more robust signal for semi-supervised learning. This approach not only improves accuracy in crowd counting but also offers a scalable solution for other object-counting tasks, such as medical cell analysis or satellite imagery, where precise localization is critical.

Summary updated

Abstract

The accuracy of crowd counting in images has improved greatly in recent years due to the development of deep neural networks for predicting crowd density maps. However, most methods do not further explore the ability to localize people in the density map, with those few works adopting simple methods, like finding the local peaks in the density map. In this paper, we propose the optimal transport minimization (OT-M) algorithm for crowd localization with density maps. The objective of OT-M is to find a target point map that has the minimal Sinkhorn distance with the input density map, and we propose an iterative algorithm to compute the solution. We then apply OT-M to generate hard pseudo-labels (point maps) for semi-supervised counting, rather than the soft pseudo-labels (density maps) used in previous methods. Our hard pseudo-labels provide stronger supervision, and also enable the use of recent density-to-point loss functions for training. We also propose a confidence weighting strategy to give higher weight to the more reliable unlabeled data. Extensive experiments show that our methods achieve outstanding performance on both crowd localization and semi-supervised counting. Code is available at https://github.com/Elin24/OT-M.
BibTeX
@inproceedings{lin2023optimal,
  author = {Wei Lin and Antoni B. Chan},
  title = {{Optimal Transport Minimization: Crowd Localization on Density Maps for Semi-Supervised Counting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02075},
}
Figure