Introduction
Object counting is a fundamental task in computer vision, typically requiring human-annotated exemplars for high performance. Zero-Shot Object Counting (ZSOC) seeks to remove this manual burden, allowing users to count objects by simply providing a class name (e.g., "pigeons"). Traditional ZSOC methods rely on a complex two-stage pipeline: first discovering exemplars based on text, then using those exemplars to count. This approach is prone to error propagation and high computational overhead.
VLCounter is a novel one-stage framework that eliminates the need for an explicit exemplar discovery stage. By leveraging the pre-trained vision-language associations of CLIP and introducing target-aware enhancements, it achieves state-of-the-art accuracy while being significantly more efficient.
Methodology
VLCounter is built upon a "Visual-Language Baseline" (VLBase) that utilizes a pre-trained CLIP model. The core idea is to treat counting as a direct vision-language alignment problem, bypassing the risks associated with picking "bad" exemplars.
Key Components
- Semantic-conditioned Prompt Tuning (SPT): Standard prompt tuning uses generic learnable tokens. SPT instead conditions these prompts on the text embedding of the target class, forcing the image encoder to produce "target-highlighted" representations from the start.
- Learnable Affine Transformation (LAT): While CLIP's similarity maps often highlight entire objects, counting models perform best when focusing on object centers. LAT is a learnable layer that refines the similarity map to better align with the needs of the counting task.
- Segment-aware Skip Connection (SaSC): To preserve the generalization power of CLIP, SaSC passes intermediate encoder features to the decoder. These features are filtered by the similarity map so the decoder only receives information relevant to the objects being counted.
The framework is optimized using a Rank-aware Contrastive Loss, encouraging the model to assign higher activation values to patches near the actual centers of objects.
Experimental Results
VLCounter was evaluated on several benchmarks, including FSC147, CARPK, and PUCPR+. It achieved a state-of-the-art Mean Absolute Error (MAE) of 17.05 on the FSC147 test set, a significant improvement over the previous best ZSOC model (MAE 22.09).
Key Findings
- Strong Generalization: In cross-dataset tests on CARPK, VLCounter achieved an MAE of 6.46, outperforming even some "few-shot" models that had access to human-annotated exemplars.
- Efficiency: By adopting a one-stage approach, VLCounter is faster at inference and roughly twice as fast to train as leading few-shot frameworks like BMNet+.
- Limitations: Qualitative analysis shows that while VLCounter handles dense scenes well, extremely high-density images (e.g., markers) still present a challenge. Additionally, the model is sensitive to prompt phrasing, with plural prompts generally yielding better results.
Conclusion
VLCounter demonstrates that the "exemplar discovery" stage in zero-shot counting is a bottleneck that can be bypassed. By treating counting as a direct vision-language alignment problem and refining CLIP's spatial information for point-based localization, VLCounter sets a new standard for efficiency and accuracy in class-agnostic object counting.
Summary updated