Single Domain Generalization for Few-Shot Counting via Universal Representation Matching

Xianing Chen, Si Huo, Borui Jiang, Hailin Hu, Xinghao Chen

Abstract

Few-shot counting (FSC) enables models to count objects of arbitrary categories using only a few labeled examples. However, existing models often struggle with domain shifts, where training and testing data come from different distributions. This paper introduces Universal Representation Matching (URM), the first framework designed for Single Domain Generalization (SDG) in few-shot counting. URM addresses the limitations of rigid, specialized prototypes by distilling "universal" vision-language knowledge from CLIP. By integrating localized visual features and LLM-enhanced linguistic descriptions, URM produces robust prototypes that generalize across diverse environments. Experimental results on FSC147 and FSCD-LVIS benchmarks demonstrate that URM achieves state-of-the-art accuracy in cross-domain scenarios without requiring target-domain adaptation.

Introduction

Few-shot counting (FSC) is a versatile task where a model counts objects of any class given a few bounding box exemplars. Despite recent progress, a significant challenge remains: domain shift. A model trained on high-quality internet images may fail on low-resolution surveillance footage or in different environmental conditions. This paper formalizes this challenge as Single Domain Generalization (SDG), where a model must generalize from a single source domain to any unseen target domain without fine-tuning.

Domain generalization for few-shot counting: training on a source domain and testing on diverse target domains without adaptation.

The authors identify a critical bottleneck in the standard "extract-then-match" pipeline. Most models extract a prototype from user examples and search for similar patterns in the image. However, narrow training data leads to rigid prototypes that fail to recognize target objects effectively in new domains. URM is proposed to overcome this by leveraging the vast, generalized knowledge of large-scale pre-trained models.

Methodology

URM enhances the counting pipeline by "borrowing" knowledge from CLIP through knowledge distillation. Unlike standard models that create prototypes solely from training examples, URM forces its internal prototypes to mimic universal vision and language representations.

Comparison of the traditional extract-then-match pipeline (b) and the proposed Universal Representation Matching (c), which incorporates CLIP-based knowledge distillation.

Key Components

  • Universal Vision Representation: Standard CLIP features are global. URM utilizes a modified MaskCLIP approach to extract localized, object-level features, ensuring the model understands individual object characteristics.
  • Universal Language Representation: Instead of simple labels, URM uses LLMs (like GPT-4) to generate rich, descriptive prompts (e.g., "a vehicle with four wheels and glass windows"). These are encoded by CLIP to provide a universal linguistic anchor for the object category.
  • Distillation during Training: During the training phase, the model's prototypes are distilled from these universal features. Crucially, the CLIP and LLM components are only used during training; at inference, the model remains a standalone, efficient counter.

Experimental Setup

The model was evaluated on two primary benchmarks: FSC147, the standard benchmark with 147 categories, and FSCD-LVIS, a more complex dataset with 372 classes used to test cross-domain generalization. The primary evaluation metric was Mean Absolute Error (MAE), focusing on performance when training on FSC147 and testing on the diverse FSCD-LVIS distribution.

Results and Analysis

URM set new records for cross-domain few-shot counting. When tested on FSCD-LVIS after training on FSC147, URM achieved an MAE of 21.87, significantly outperforming the previous state-of-the-art, MPCount (MAE 25.11).

Key insights from the evaluation include:

  • Generalization vs. Specialization: t-SNE visualizations reveal that while vanilla models produce overlapping clusters for unseen categories, URM maintains clear boundaries, avoiding the "narrow distribution" trap.
  • Vision-Language Synergy: The authors found that combining vision and language representations is optimal, with a 90% weight toward vision distillation yielding the best results. Language provides the concept, while vision provides the texture and shape.
  • Zero-Shot Capability: Even without visual exemplars, URM outperformed specialized models by counting based solely on category names.

Conclusion

URM represents a significant advancement in few-shot counting by introducing the first solution for single domain generalization. By distilling universal representations from CLIP, the model achieves robust performance across unseen domains. While the model currently depends on category names and involves training complexity due to multi-model distillation, it paves the way for future open-vocabulary counting systems that can operate based on complex natural language queries.

Summary updated

Abstract

Few-shot counting estimates the number of target objects in an image using only a few annotated exemplars. However, domain shift severely hinders existing methods to generalize to unseen scenarios. This falls into the realm of single domain generalization that remains unexplored in few-shot counting. To solve this problem, we begin by analyzing the main limitations of current methods, which typically follow a standard pipeline that extract the object prototypes from exemplars and then match them with image feature to construct the correlation map. We argue that existing methods overlook the significance of learning highly generalized prototypes. Building on this insight, we propose the first single domain generalization few-shot counting model, Universal Representation Matching, termed URM. Our primary contribution is the discovery that incorporating universal vision-language representations distilled from a large scale pretrained vision-language model into the correlation construction process substantially improves robustness to domain shifts without compromising in domain performance. As a result, URM achieves state-of-the-art performance on both in domain and the newly introduced domain generalization setting.
BibTeX
@inproceedings{chen2025single,
  author = {Xianing Chen and Si Huo and Borui Jiang and Hailin Hu and Xinghao Chen},
  title = {{Single Domain Generalization for Few-Shot Counting via Universal Representation Matching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00437},
}
Figure