Single Domain Generalization for Few-Shot Counting via Universal Representation Matching

Xianing Chen, Si Huo, Borui Jiang, Hailin Hu, Xinghao Chen

Abstract

Few-shot counting (FSC) models typically struggle to generalize to new, unseen environments because they learn overly narrow object prototypes from their training data. We introduce Universal Representation Matching (URM), a novel approach that distills broad vision-language knowledge from a frozen CLIP model into the model's prototypes during training. By matching image features against these "universal" prototypes, URM achieves state-of-the-art performance in both standard and newly introduced cross-domain counting tasks. Crucially, URM maintains inference efficiency by only requiring distillation during the training phase.

Introduction

Few-shot counting (FSC) aims to estimate the number of target objects in an image using only a few annotated examples (exemplars) or none at all (zero-shot). Current state-of-the-art models follow an "extract-then-match" paradigm: they extract a visual prototype from the exemplars and match it against the rest of the image to find similar objects.

However, these models suffer severe performance drops when deployed in the real world due to domain shift—changes in camera angles, weather, or object types. Because they are trained on a single source domain, the learned prototypes have a narrow distribution. This paper tackles the challenge of single domain generalization in FSC: training a model on one domain so that it can accurately count objects in completely unseen target domains without requiring any retraining or target data.

Domain generalization for few-shot counting: the challenge of transferring knowledge from a source domain to an unseen target domain without adaptation.

Universal Representation Matching (URM)

The authors propose Universal Representation Matching (URM) to inject broad, open-world knowledge into object prototypes during training. This ensures they generalize better to unseen domains.

Key Components

  • Universal Vision Prototypes: Instead of just learning visual features from the limited training set, URM uses a frozen CLIP model as a teacher. Since standard CLIP focuses on global image properties, the authors use a modified MaskCLIP approach to extract dense, localized object-level representations. These are distilled into the model's vision prototypes via a feature mimicry loss.
  • Universal Language Prototypes: Large Language Models (LLMs like GPT-4) generate diverse, descriptive text prompts about target object categories. These prompts are encoded by CLIP's text encoder and distilled into the model's language prototypes.
  • Cross-Attention Matching: The learned vision and language prototypes are concatenated and matched against target image features using cross-attention layers to generate a correlation map, which is then regressed into a final 2D density map.
Comparison of the standard (b) extract-then-match paradigm versus the proposed (c) universal representation matching, which incorporates CLIP-based distillation.

Experimental Setup

The model was evaluated on standard benchmarks including FSC-147 (6,135 images, 147 categories) and FSCD-LVIS (6,195 images, 372 classes). To explicitly test single domain generalization, the authors introduced a cross-domain evaluation setting (FSC-147 → FSCD-LVIS and vice versa), where the model is trained entirely on one dataset and tested on the other.

Key Insights and Results

  • Overcoming Prototype Bottlenecks: t-SNE visualizations reveal that standard models learn prototypes with highly constrained, overlapping feature spaces. Broadening this distribution via CLIP creates well-defined, open-set visual concept boundaries.
  • Complementary Modalities: Ablation studies show that using both visual and textual distillation works significantly better than either alone. Language provides high-level semantic boundaries, while vision provides fine-grained local details.
  • State-of-the-Art Performance: On the FSC-147 → FSCD-LVIS task, URM achieved a Mean Absolute Error (MAE) of 21.87, significantly outperforming the previous best domain generalization method (MPCount, MAE 25.11). In zero-shot settings, URM dropped MAE from 27.68 to 23.54.
  • In-Domain Excellence: Despite its focus on robustness, URM also improved in-domain performance on FSC-147, achieving an MAE of 10.63 (and 8.47 with the DAVE verification module), surpassing the previous SOTA (LOCA at 10.24).

Conclusion and Future Work

URM demonstrates that distilling external vision-language knowledge specifically into prototypes can effectively bridge the gap between training and real-world deployment for few-shot counting. While the method relies on CLIP's quality and category names for prompting, the integration of Large Vision-Language Models (LVLMs) shows promise for fully automated, open-world counting. Future work will explore more efficient local feature extraction from larger foundation models.

Summary updated

BibTeX
@inproceedings{chen2025single,
  author = {Xianing Chen and Si Huo and Borui Jiang and Hailin Hu and Xinghao Chen},
  title = {{Single Domain Generalization for Few-Shot Counting via Universal Representation Matching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00437},
}
Figure