Abstract
Semi-supervised crowd counting is crucial for addressing the high annotation costs of densely populated scenes. Although several methods based on pseudo-labeling have been proposed, it remains challenging to effectively and accurately utilize unlabeled data. In this paper, we propose a novel framework called Taste More Taste Better (TMTB), which emphasizes both data and model aspects. Firstly, we explore a data augmentation technique well-suited for the crowd counting task. By inpainting the background regions, this technique can effectively enhance data diversity while preserving the fidelity of the entire scenes. Secondly, we introduce the Visual State Space Model as backbone to capture the global context information from crowd scenes, which is crucial for extremely crowded, low-light, and adverse weather scenarios. In addition to the traditional regression head for exact prediction, we employ an Anti-Noise classification head to provide less exact but more accurate supervision, since the regression head is sensitive to noise in manual annotations. We conduct extensive experiments on four benchmark datasets and show that our method outperforms state-of-the-art methods by a large margin.
Introduction
Crowd counting—the task of estimating the number of individuals in dense images—is a cornerstone of modern urban planning and public safety. However, the development of high-performing models is hindered by the immense cost of manual annotation. For instance, annotating a single large-scale dataset like UCF-QNRF can require upwards of 2,000 human hours of meticulous point-clicking.
Semi-supervised crowd counting (SSCC) offers a promising path forward by leveraging small amounts of labeled data alongside vast unlabeled pools. Yet, existing SSCC methods often struggle with two key issues: ineffective data augmentation that disrupts spatial structures and model architectures that fail to capture global scene context. In this paper, we present the Taste More, Taste Better (TMTB) framework, a novel approach that addresses these bottlenecks through high-fidelity data augmentation and a state-of-the-art Visual State Space Model (VSSM) backbone.
The TMTB Framework
The TMTB framework is built on the philosophy that both the quality of data ("Taste More") and the strength of the model ("Taste Better") are essential for effective semi-supervised learning. The framework utilizes a Mean Teacher architecture where a "Teacher" model generates pseudo-labels for unlabeled data to train a "Student" model, enhanced by several key innovations.
1. Inpainting Augmentation: Preserving Spatial Integrity
Traditional data augmentation techniques like Mixup or CutMix, while effective for general image classification, often fail in crowd counting. These methods can destroy the spatial logic of a scene, creating "impossible" density maps that confuse the model.
To solve this, TMTB employs Inpainting Augmentation. Using a diffusion-based model (Stable Diffusion), the framework "inpaints" the background regions of crowd scenes. By keeping the foreground (the people) intact while varying the environmental context, the model generates diverse, high-fidelity training samples. This ensures that the spatial relationships necessary for accurate density estimation remain preserved while the model learns to generalize across different backgrounds. To further ensure quality, the system includes a weighted mask to filter out poorly inpainted regions where the Teacher and Student models disagree significantly.
2. VSSM Backbone: Capturing Global Context
Crowd scenes are often characterized by extreme density, low lighting, or adverse weather. Standard Convolutional Neural Networks (CNNs) tend to focus on local features and may struggle with these global environmental challenges. While Transformers can capture long-range dependencies, their computational cost is often prohibitive for high-resolution crowd images.
TMTB introduces the Visual State Space Model (VSSM), specifically the VMamba architecture, as its backbone. VSSM provides the global receptive field of a Transformer with the linear computational complexity of a CNN. This allows the model to reason about the entire scene—understanding, for example, that a dark, blurry corner in a stadium likely contains people—without the overhead of traditional attention mechanisms.
3. Anti-Noise Classification Head
Manual crowd annotations are notoriously noisy; points are often slightly misplaced or individuals are missed entirely. Because regression heads are highly sensitive to such noise, TMTB adds an auxiliary Anti-Noise classification head. This branch predicts "count bins" (e.g., categorizing a region as having a low, medium, or high density) rather than an exact number. This provides a stable, robust supervision signal that guides the more sensitive regression head, acting as a "safety net" against imperfect ground-truth labels.
Experimental Results
The TMTB framework was evaluated on four major benchmarks: JHU-Crowd++, UCF-QNRF, ShanghaiTech Part A, and ShanghaiTech Part B. The experiments focused on low-label regimes (5%, 10%, and 40% labeling) to prove the model's efficiency with minimal data.
- JHU-Crowd++: At just 5% labels, TMTB achieved a Mean Absolute Error (MAE) of 67.0, a massive improvement over the previous state-of-the-art (SOTA) of 76.5. This performance even surpassed some models trained on 10% labels.
- UCF-QNRF: TMTB reduced the MAE to 96.3 at 5% labels, surpassing the previous best of 101.4.
- Generalization: In cross-dataset testing (Domain Generalization), TMTB demonstrated superior "zero-shot" performance, indicating that the combination of inpainting and the VSSM backbone helps the model learn fundamental crowd features rather than dataset-specific noise.
Conclusion and Future Work
The "Taste More, Taste Better" framework represents a significant step forward for semi-supervised crowd counting. By prioritizing spatial integrity through inpainting and global context through VSSM, TMTB achieves state-of-the-art performance with minimal human intervention.
While the results are promising, the authors note that the inpainting process can occasionally generate fragmented human features in the background. Future work will focus on refining the fidelity of generated humans and exploring ways to integrate the framework without the complexity of a pre-trained diffusion model.
Summary updated