GCNet: Probing Self-Similarity Learning for Generalized Counting Network

Mingjie Wang, Yande Li, Jun Zhou, Graham W. Taylor, Minglun Gong

TL;DR

This paper introduces GCNet, a novel class-agnostic counting network that eliminates the need for user-provided exemplar annotations and density maps. By learning self-similarity patterns from inherent repetitions within images, GCNet achieves state-of-the-art performance on exemplar-free counting tasks and demonstrates strong generalization to unseen object categories and crowd-counting datasets.

What Problem Does This Paper Address?

The task of Class-Agnostic Counting (CAC) aims to count objects of any category without prior knowledge of their specific class. Existing CAC methods often rely on user-provided "exemplars" (example object patches) and detailed "density maps" (pixel-level annotations indicating object locations) for training. This reliance creates several limitations:

  • Limited Generality: Requiring exemplars and density maps makes these models difficult to apply in real-world "plug-and-play" scenarios where such annotations are unavailable or impractical to obtain.
  • Laborious Annotation: Creating density maps is time-consuming and labor-intensive, hindering the scalability of these methods.
  • Suboptimal Performance: The quality and diversity of provided exemplars can bound the performance of matching-based approaches.

GCNet aims to overcome these challenges by developing a truly "exemplar-free" and "weakly-supervised" counting framework that can generalize to a wide range of object categories and scenes.

Methods

GCNet introduces several key innovations to achieve exemplar-free and weakly-supervised counting:

  • Pseudo Exemplar Simulator: Instead of relying on user-provided exemplars, GCNet uses a pseudo-Siamese structure to automatically learn "pseudo exemplar clues" from inherent repetition patterns within the input image. This simulator extracts discriminative representations without explicit exemplar annotations.
  • Self-Similarity Learning: The core of GCNet is its "self-similarity learning" strategy. It leverages a "Dual-Attention Self-Similarity (DASS)" learning protocol to construct a high-fidelity similarity map. This involves:
  • Anisotropic Encoder: This component captures discriminative features by using kernels oriented along horizontal, vertical, and channel directions, allowing it to handle objects with varied shapes and orientations.
  • Dual-Attention Condenser: This module refines features by distilling knowledge across different branches (pseudo exemplar and main image features) and using attention mechanisms to emphasize informative directions.
  • Weakly-Supervised Location-Aware Counter: GCNet eliminates the need for density maps by training with only count-level supervisory signals (i.e., the total number of objects in an image). It uses a regression head to directly predict the final count from the learned similarity map and image features. This approach mimics human subitizing behavior, which estimates set sizes without precise localization.

Datasets

The paper evaluates GCNet on the following datasets:

  • FSC147: This is a large-scale dataset specifically designed for class-agnostic counting, featuring 147 object categories. It was used for training and evaluating the core exemplar-free and weakly-supervised capabilities.
  • Crowd-Specific Datasets: To demonstrate generalization, GCNet was also tested on three popular crowd-counting datasets:
  • ShanghaiTech Part A
    • ShanghaiTech Part B
    • UCF_QNRF

Key Insights and Ideas

  • Self-Similarity as a Universal Cue: The paper demonstrates that inherent self-similarity patterns within an image are a powerful and generalizable cue for counting, even across diverse object categories. This insight allows for the elimination of class-specific exemplars.
  • Weak Supervision is Sufficient: By moving from dense pixel-level annotations to simple count-level labels, GCNet significantly reduces annotation burden while maintaining high performance. This highlights the potential of weak supervision for complex vision tasks.
  • Anisotropic Features for Robustness: The use of anisotropic features in the DASS module is crucial for handling the variability in object shapes and orientations commonly found in real-world images, leading to more robust similarity learning.
  • Exemplar-Free is Superior for Generality: The experiments suggest that a truly exemplar-free approach, even when compared to an exemplar-guided version of GCNet, can lead to better generalization, possibly by preventing the model from being overly biased by specific exemplar examples.

Key Results

  • State-of-the-Art on FSC147: GCNet achieves state-of-the-art results in the exemplar-free CAC category on the FSC147 benchmark, outperforming existing methods like RepRPN-Counter.
  • Competitive Performance: GCNet performs on par with or better than exemplar-dependent methods on FSC147, despite not using any exemplar information during inference.
  • Strong Cross-Dataset Generalization: GCNet shows impressive performance on crowd-counting datasets (ShanghaiTech A/B, UCF_QNRF) even when trained on FSC147, which has very few crowd scenes. This highlights its strong transferability.
  • Ablation Studies: The paper's ablation studies confirm the effectiveness of each proposed component: the pseudo exemplar simulator, the dual-attention self-similarity learning, and the weakly-supervised counter.

Limitations and Future Work

  • Counting All Repeating Instances: GCNet, by design, tends to count all repeating instances it detects, which can lead to confusion when objects from different categories have similar appearances (e.g., counting strawberries and blueberries together). The authors acknowledge this limitation.
  • Future Work - Visual Grounding: The authors propose incorporating visual grounding techniques in the future. This would allow users to specify what to count using keywords or points of interest, providing more flexibility and control without the need for explicit exemplars. This could help address the issue of counting mixed categories.

Summary updated

Abstract

The class-agnostic counting (CAC) problem has caught increasing attention recently due to its wide societal applications and arduous challenges. To count objects of different categories, existing approaches rely on user-provided exemplars, which is hard-to-obtain and limits their generality. In this paper, we aim to empower the framework to recognize adaptive exemplars within the whole images. A zero-shot Generalized Counting Network (GCNet) is developed, which uses a pseudo-Siamese structure to automatically and effectively learn pseudo exemplar clues from inherent repetition patterns. In addition, a weakly-supervised scheme is presented to reduce the burden of laborious density maps required by all contemporary CAC models, allowing GCNet to be trained using count-level supervisory signals in an end-to-end manner. Without providing any spatial location hints, GCNet is capable of adaptively capturing them through a carefully-designed self-similarity learning strategy. Extensive experiments and ablation studies on the prevailing benchmark FSC147 for zero-shot CAC demonstrate the superiority of our GCNet. It performs on par with existing exemplar-dependent methods and shows stunning cross-dataset generality on crowd-specific datasets, e.g., ShanghaiTech Part A, Part B and UCF_QNRF.
BibTeX
@misc{wang2023gcnet,
  author = {Mingjie Wang and Yande Li and Jun Zhou and Graham W. Taylor and Minglun Gong},
  title = {{GCNet: Probing Self-Similarity Learning for Generalized Counting Network}},
  year = {2023},
  eprint = {2302.05132},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.05132},
}

Loading related content...

Figure