Class Agnostic Object Counting

25 concepts25 papers2 proposals

Traditional visual counting models are specialists. As noted by Gian Gao et al. in their overview of crowd counting architectures, deep learning models have historically been hard-coded for specific object classes, such as human heads or vehicles. If you wanted to count cells under a microscope, you had to collect a specialized dataset of cells, annotate them with dot-annotations, and train a brand-new model from scratch.

Class-agnostic object counting fundamentally re-frames this problem. Instead of asking a network to learn what a specific object looks like, it trains the network to understand repetition and similarity. The goal is a universal counting machine: a single model that can enumerate any arbitrary object category in the wild—whether it is penguins, cars, or timber logs—given only a few visual examples or a textual prompt.

The Core Mechanism: From Classification to Similarity Matching

To count objects without knowing their semantic category in advance, models must shift from category-specific regression to dense feature matching.

In a typical few-shot setup, the user provides a query image and a few bounding boxes containing examples of the target object (called exemplars). Early formulations of this paradigm, such as work by Erika Lu et al. in Class-Agnostic Counting and Viresh Ranjan et al. in Learning To Count Everything, established a foundational pipeline:

  1. Extract feature maps from both the query image and the exemplars using a shared backbone network.
  2. Compute a similarity map (often via cross-correlation) between the exemplar features and the query image features.
  3. Pass this similarity map through a regression network to output a dense density map, where integrating over the map yields the final count.
[Query Image]    --> [Feature Extractor] ---\
                                             --> [Similarity Matching] --> [Density Regressor] --> Final Count
[Exemplar Box]   --> [Feature Extractor] ---/

The intuition is straightforward: if the exemplar feature vector strongly correlates with a patch in the query image, that patch likely contains an instance of the object.

To make this matching process robust, subsequent architectures focused on enhancing how features are compared. For instance, Min Shi et al. developed BMNet+ in Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting, replacing fixed similarity metrics with learnable bilinear matching modules that explicitly supervise the similarity map generation.

Expanding on this, Zhiyuan You et al. introduced SAFECount in Few-shot Object Counting with Similarity-Aware Feature Enhancement, which iteratively fuses the computed similarity maps back into the image features. This reinforcement loop ensures that the network dynamically emphasizes target-specific patterns while suppressing background noise.

To capture finer geometric nuances, Yuanwu Xu proposed the Spatial Similarity Distribution (SSD) network in Learning Spatial Similarity Distribution for Few-shot Object Counting. Instead of collapsing exemplar features into a single vector, the author preserves their full spatial structure by calculating a 4D similarity pyramid. This allows the model to distinguish complex matching patterns, such as the structural difference between an object’s center and its boundaries.

Overcoming Intraclass Diversity and Scale

In real-world scenarios, objects belonging to the same class are rarely identical copies. A single image might contain cars of different colors, orientations, and scales, or items that are partially occluded. If a model matches strictly against a single visual exemplar, it will fail to count instances that deviate from that specific example.

The field addresses this "intraclass diversity gap" through two primary strategies: feature hallucination and iterative prototype adaptation.

In Class-Agnostic Object Counting Robust to Intraclass Diversity, Shenjian Gong et al. introduced the Robust Class-Agnostic Counter (RCAC). RCAC mathematically "hallucinates" variations of the user-provided exemplars directly within the feature space, simulating changes in scale, orientation, and appearance to broaden the search criteria.

Alternatively, N Djukić et al. developed LOCA in A Low-Shot Object Counting Network With Iterative Prototype Adaptation. Rather than relying on static exemplar features, LOCA uses the broader image context to iteratively adapt the target "prototype." By looking at the entire scene, the model refines its understanding of what constitutes an object instance in that specific environment, dynamically accommodating shape and appearance variations.

To capture long-range dependencies and global context across the entire image, Andrew Zisserman et al. introduced CounTR in CounTR: Transformer-based Generalised Visual Counting. By employing a vision transformer architecture combined with a specialized mosaic data augmentation pipeline, CounTR explicitly models the global relationships between all image patches, preventing the network from losing track of objects in highly congested or varied scenes.

Eliminating the Exemplar: Zero-Shot and Text-Prompted Counting

While few-shot counting reduces the need for category-specific training, requiring users to manually draw bounding boxes at test time remains a significant operational bottleneck. This realization catalyzed a shift toward zero-shot object counting, where a user specifies what to count using only natural language descriptions or text labels.

The core challenge of text-prompted counting is bridging the gap between textual concepts and spatial visual features. To solve this, researchers have increasingly turned to vision-language foundation models and generative AI.

In Zero-Shot Object Counting, Hieu Le et al. demonstrated that a text label can be converted into a semantic prototype. This prototype interacts with the image features to automatically localize the best internal visual exemplars within the scene, effectively converting a zero-shot text query into an automated few-shot visual matching task.

Huilin Zhu et al. optimized this selection pipeline in VA-Count (Zero-Shot Object Counting with Good Exemplars) by introducing a contrastive filtering system that evaluates and selects only the highest-quality internal exemplars, drastically reducing counting errors stemming from ambiguous text prompts.

[Text Prompt: "apples"] --> [Semantic Prototype] --> [Internal Exemplar Selection] --> [Standard Matching Pipeline]

Other approaches directly exploit the rich spatial-semantic representations embedded within pre-trained generative models. Xiaofei Hui et al. developed CountDiff in Class-Agnostic Object Counting with Text-to-Image Diffusion Model. Instead of training a new matching network, CountDiff extracts semantic information directly from the internal cross-attention maps of a pre-trained text-to-image diffusion model (like Stable Diffusion), repurposing its generative "world knowledge" to locate and count arbitrary objects.

To provide maximum flexibility, Niki Amini-Naieni et al. introduced CountGD in CountGD: Multi-Modal Open-World Counting. By adapting the GroundingDINO detection foundation model, CountGD accepts text prompts, visual exemplars, or any combination of both, creating a unified framework for open-world multi-modal queries.

Taking autonomy to its logical conclusion, Michael A. Hobley proposed ABC123 in ABC Easy as 123: A Blind Counter for Exemplar-Free Multi-Class Class-Agnostic Counting. This architecture operates completely "blindly"—requiring neither visual exemplars nor text prompts—and is capable of simultaneously isolating, categorizing, and counting multiple distinct object classes within a single scene without user intervention.

Training-Free and Weakly Supervised Paradigms

As the field matures, researchers are actively trying to bypass the need for expensive, manual point-by-point annotations during the model's own training phase.

One direction is completely training-free inference. In Training-free Object Counting with Prompts, Zenglin Shi et al. demonstrated that by combining the zero-shot segmentation capabilities of foundation models like SAM (Segment Anything Model) with the semantic grounding of CLIP, one can achieve highly accurate object counts out-of-the-box without executing a single gradient update for the counting task.

Another direction leverages weak supervision or synthetic data generation:

By shifting from rigid, category-specific detectors to flexible, similarity-driven foundation models, class-agnostic counting has transformed visual enumeration from a series of isolated engineering tasks into a unified, open-world capability.

Key Foundational Work

From Open Set to Closed Set: Supervised Spatial Divide-and-Conquer for Object Counting (2020) — contributed 2 concept elements to this direction.
Class-Agnostic Counting (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_class_agnostic_generalization_gap_ebc70962,
  title = {class-agnostic generalization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-agnostic-generalization-gap/rdr_ebc70962771e47cebf647ab8e8959265},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_class_agnostic_generalization_gap_ebc70962_v1,
  title = {class-agnostic generalization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-agnostic-generalization-gap/rdr_ebc70962771e47cebf647ab8e8959265/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake