Abstract
Few-shot counting (FSC) enables models to count objects of arbitrary categories using only a few labeled examples. However, existing models often struggle with domain shifts, where training and testing data come from different distributions. This paper introduces Universal Representation Matching (URM), the first framework designed for Single Domain Generalization (SDG) in few-shot counting. URM addresses the limitations of rigid, specialized prototypes by distilling "universal" vision-language knowledge from CLIP. By integrating localized visual features and LLM-enhanced linguistic descriptions, URM produces robust prototypes that generalize across diverse environments. Experimental results on FSC147 and FSCD-LVIS benchmarks demonstrate that URM achieves state-of-the-art accuracy in cross-domain scenarios without requiring target-domain adaptation.
Introduction
Few-shot counting (FSC) is a versatile task where a model counts objects of any class given a few bounding box exemplars. Despite recent progress, a significant challenge remains: domain shift. A model trained on high-quality internet images may fail on low-resolution surveillance footage or in different environmental conditions. This paper formalizes this challenge as Single Domain Generalization (SDG), where a model must generalize from a single source domain to any unseen target domain without fine-tuning.
The authors identify a critical bottleneck in the standard "extract-then-match" pipeline. Most models extract a prototype from user examples and search for similar patterns in the image. However, narrow training data leads to rigid prototypes that fail to recognize target objects effectively in new domains. URM is proposed to overcome this by leveraging the vast, generalized knowledge of large-scale pre-trained models.
Methodology
URM enhances the counting pipeline by "borrowing" knowledge from CLIP through knowledge distillation. Unlike standard models that create prototypes solely from training examples, URM forces its internal prototypes to mimic universal vision and language representations.
Key Components
- Universal Vision Representation: Standard CLIP features are global. URM utilizes a modified MaskCLIP approach to extract localized, object-level features, ensuring the model understands individual object characteristics.
- Universal Language Representation: Instead of simple labels, URM uses LLMs (like GPT-4) to generate rich, descriptive prompts (e.g., "a vehicle with four wheels and glass windows"). These are encoded by CLIP to provide a universal linguistic anchor for the object category.
- Distillation during Training: During the training phase, the model's prototypes are distilled from these universal features. Crucially, the CLIP and LLM components are only used during training; at inference, the model remains a standalone, efficient counter.
Experimental Setup
The model was evaluated on two primary benchmarks: FSC147, the standard benchmark with 147 categories, and FSCD-LVIS, a more complex dataset with 372 classes used to test cross-domain generalization. The primary evaluation metric was Mean Absolute Error (MAE), focusing on performance when training on FSC147 and testing on the diverse FSCD-LVIS distribution.
Results and Analysis
URM set new records for cross-domain few-shot counting. When tested on FSCD-LVIS after training on FSC147, URM achieved an MAE of 21.87, significantly outperforming the previous state-of-the-art, MPCount (MAE 25.11).
Key insights from the evaluation include:
- Generalization vs. Specialization: t-SNE visualizations reveal that while vanilla models produce overlapping clusters for unseen categories, URM maintains clear boundaries, avoiding the "narrow distribution" trap.
- Vision-Language Synergy: The authors found that combining vision and language representations is optimal, with a 90% weight toward vision distillation yielding the best results. Language provides the concept, while vision provides the texture and shape.
- Zero-Shot Capability: Even without visual exemplars, URM outperformed specialized models by counting based solely on category names.
Conclusion
URM represents a significant advancement in few-shot counting by introducing the first solution for single domain generalization. By distilling universal representations from CLIP, the model achieves robust performance across unseen domains. While the model currently depends on category names and involves training complexity due to multi-model distillation, it paves the way for future open-vocabulary counting systems that can operate based on complex natural language queries.
Summary updated