VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting

Seunggu Kang, WonJun Moon, Euiyeon Kim, Jae-Pil Heo

Introduction

Object counting is a fundamental task in computer vision, typically requiring human-annotated exemplars for high performance. Zero-Shot Object Counting (ZSOC) seeks to remove this manual burden, allowing users to count objects by simply providing a class name (e.g., "pigeons"). Traditional ZSOC methods rely on a complex two-stage pipeline: first discovering exemplars based on text, then using those exemplars to count. This approach is prone to error propagation and high computational overhead.

Comparison of the proposed one-stage VLCounter framework against traditional two-stage exemplar-based methods.

VLCounter is a novel one-stage framework that eliminates the need for an explicit exemplar discovery stage. By leveraging the pre-trained vision-language associations of CLIP and introducing target-aware enhancements, it achieves state-of-the-art accuracy while being significantly more efficient.

Methodology

VLCounter is built upon a "Visual-Language Baseline" (VLBase) that utilizes a pre-trained CLIP model. The core idea is to treat counting as a direct vision-language alignment problem, bypassing the risks associated with picking "bad" exemplars.

The VLCounter architecture, illustrating the integration of Semantic-conditioned Prompt Tuning (SPT), Learnable Affine Transformation (LAT), and Segment-aware Skip Connection (SaSC).

Key Components

  • Semantic-conditioned Prompt Tuning (SPT): Standard prompt tuning uses generic learnable tokens. SPT instead conditions these prompts on the text embedding of the target class, forcing the image encoder to produce "target-highlighted" representations from the start.
  • Learnable Affine Transformation (LAT): While CLIP's similarity maps often highlight entire objects, counting models perform best when focusing on object centers. LAT is a learnable layer that refines the similarity map to better align with the needs of the counting task.
  • Segment-aware Skip Connection (SaSC): To preserve the generalization power of CLIP, SaSC passes intermediate encoder features to the decoder. These features are filtered by the similarity map so the decoder only receives information relevant to the objects being counted.

The framework is optimized using a Rank-aware Contrastive Loss, encouraging the model to assign higher activation values to patches near the actual centers of objects.

Experimental Results

VLCounter was evaluated on several benchmarks, including FSC147, CARPK, and PUCPR+. It achieved a state-of-the-art Mean Absolute Error (MAE) of 17.05 on the FSC147 test set, a significant improvement over the previous best ZSOC model (MAE 22.09).

Key Findings

  • Strong Generalization: In cross-dataset tests on CARPK, VLCounter achieved an MAE of 6.46, outperforming even some "few-shot" models that had access to human-annotated exemplars.
  • Efficiency: By adopting a one-stage approach, VLCounter is faster at inference and roughly twice as fast to train as leading few-shot frameworks like BMNet+.
  • Limitations: Qualitative analysis shows that while VLCounter handles dense scenes well, extremely high-density images (e.g., markers) still present a challenge. Additionally, the model is sensitive to prompt phrasing, with plural prompts generally yielding better results.

Conclusion

VLCounter demonstrates that the "exemplar discovery" stage in zero-shot counting is a bottleneck that can be bypassed. By treating counting as a direct vision-language alignment problem and refining CLIP's spatial information for point-based localization, VLCounter sets a new standard for efficiency and accuracy in class-agnostic object counting.

Summary updated

Abstract

Zero-Shot Object Counting (ZSOC) aims to count referred instances of arbitrary classes in a query image without human-annotated exemplars. To deal with ZSOC, preceding studies proposed a two-stage pipeline: discovering exemplars and counting. However, there remains a challenge of vulnerability to error propagation of the sequentially designed two-stage process. In this work, an one-stage baseline, Visual-Language Baseline (VLBase), exploring the implicit association of the semantic-patch embeddings of CLIP is proposed. Subsequently, the extension of VLBase to Visual-language Counter (VLCounter) is achieved by incorporating three modules devised to tailor VLBase for object counting. First, Semantic-conditioned Prompt Tuning (SPT) is introduced within the image encoder to acquire target-highlighted representations. Second, Learnable Affine Transformation (LAT) is employed to translate the semantic-patch similarity map to be appropriate for the counting task. Lastly, the layer-wisely encoded features are transferred to the decoder through Segment-aware Skip Connection (SaSC) to keep the generalization capability for unseen classes. Through extensive experiments on FSC147, CARPK, and PUCPR+, the benefits of the end-to-end framework, VLCounter, are demonstrated.
BibTeX
@inproceedings{kang2024vlcounter,
  author = {Seunggu Kang and WonJun Moon and Euiyeon Kim and Jae-Pil Heo},
  title = {{VLCounter: Text-Aware Visual Representation for Zero-Shot Object Counting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.28050},
}
Figure