SQLNet: Scale-Modulated Query and Localization Network for Few-Shot Class-Agnostic Counting

Hefeng Wu, Yandong Chen, Lingbo Liu, Tianshui Chen, Keze Wang, Liang Lin

TL;DR

SQLNet (Scale-Modulated Query and Localization Network) shifts the paradigm of few-shot class-agnostic counting from blurry density map regression to precise object localization. By explicitly predicting both the location and approximate size of objects based on just a few examples, it achieves state-of-the-art counting accuracy while providing the bounding boxes needed for real-world downstream tasks.

What Problem Does This Paper Address?

Imagine you need to count something specific—say, a particular type of cell in a microscope image or a specific car model in a parking lot—but you only have three examples of what that object looks like. This is Class-Agnostic Counting (CAC).

Most current top-tier methods solve this using density map regression. They produce a heat map where the "brightness" of a pixel represents object density, and the total count is the sum of all pixel values. While effective for counting, this approach has two major flaws:

  1. Lack of Precision: It doesn't tell you exactly where the objects are or how big they are, making it useless for tasks like tracking or detailed analysis.
  2. Wasted Information: These models often ignore the scale information (the size of the example bounding boxes) provided by the user, which is a crucial hint for finding similar objects.

SQLNet addresses these by treating counting as a localization problem, similar to object detection but tailored for the "few-shot" scenario where you don't have thousands of labeled training images for every possible class.

Methods

SQLNet introduces a three-stage pipeline designed to squeeze every bit of information out of the provided exemplars:

  1. Hierarchical Exemplars Collaborative Enhancement (HECE): Instead of looking at exemplars in isolation, this module lets them "talk" to each other using a Transformer-like self-attention mechanism. It also introduces Equifrequent Size Prompts (ESP)—learnable embeddings that encode the width and height of the exemplars, helping the model understand the target scale from the start.
  2. Exemplars-Unified Query Correlation (EUQC): This module performs a "unified" search. It compares the entire query image against the enhanced exemplar features all at once, using both spatial and channel-wise correlation to highlight potential matches.
  3. Scale-aware Multi-head Localization (SAML): The final stage predicts three things for every potential object: a confidence score, a precise location offset, and an approximate size.
The SQLNet architecture overview, showing the HECE, EUQC, and SAML modules.

To train this effectively without full bounding box annotations for every object (since CAC datasets usually only provide points), the authors use a Scale-aware Localization (SAL) Loss. It uses the Hungarian algorithm to dynamically match predicted points to ground-truth points and uses the user-provided exemplar sizes to supervise the size predictions of the found objects.

Datasets

The authors evaluated SQLNet on several key benchmarks:

  • FSC-147: The standard large-scale benchmark for CAC, containing over 6,000 images across 147 diverse classes. It is an "open-set" challenge, meaning the classes used for testing are never seen during training.
  • CARPK: A specialized dataset for car counting in parking lots, used here to test how well the model generalizes to new environments (cross-dataset evaluation).
  • Val-COCO & Test-COCO: Subsets of the FSC-147 dataset used to compare SQLNet against traditional object detection models like Faster R-CNN.

Key Insights and Ideas

  • Localization is a Viable Alternative to Regression: For years, the community assumed regression was the only way to get high accuracy in CAC. SQLNet proves that a localization-based approach can actually be more accurate while providing much richer output.
  • Scale as a First-Class Citizen: By treating object size as a supervised signal (derived from the exemplars), the model becomes much more robust. The "Size Prompt" acts like a mental shortcut, telling the model exactly what scale to look for.
  • Unified Interaction Beats One-by-One: Most models compare the image to Exemplar A, then Exemplar B, then Exemplar C, and average the results. SQLNet treats all exemplars as a single "class representation," which is both more computationally efficient and better at capturing the common features of the target class.
Fig. 1. Illustration for the comparison between our method and density map regression based methods. All previous state-of-the-art CAC methods follow a density map regression scheme. Top row: The left image shows the ground truth, where each targeted object is given by a point. The right image is a ground-truth density map generated from the points. The input is a raw image with several exemplars’ bounding boxes given. Bottom row: The left image shows the result of our method, predicting each object’s location along with its size. The right image shows the result of the leading method SAFECount [3]. The top-right corner of an image exhibits the counting number.

Key Results

SQLNet sets a new high bar for the field:

  • Counting Accuracy: On the FSC-147 test set, SQLNet achieved a Mean Absolute Error (MAE) of 12.49, significantly beating the previous state-of-the-art, SAFECount (14.32), and regression-based SPDCN (13.51).
  • Generalization: When trained on FSC-147 and tested directly on the CARPK car-counting dataset without any fine-tuning, SQLNet achieved an MAE of 7.66, outperforming models that were actually trained on CARPK.
  • Efficiency: Despite its complexity, SQLNet is fast. It processes images in roughly 26ms, making it over 7x faster than SAFECount (188ms).

Limitations and Future Work

While SQLNet is a major step forward, the authors note a few hurdles:

  • Multi-class Scenes: The model is currently optimized to count one type of object at a time. In images containing multiple different classes (e.g., counting both dogs and cats simultaneously), performance is "acceptable but not good enough."
  • Extreme Crowds and Occlusion: In scenarios with massive occlusion (like horses running in a dust cloud) or extremely tiny, dense objects (like distant flocks of birds), the model still misses objects.

Future work will likely focus on collecting more diverse multi-class training data and refining the localization heads to handle even higher object densities.

Summary updated

Abstract

The class-agnostic counting (CAC) task has recently been proposed to solve the problem of counting all objects of an arbitrary class with several exemplars given in the input image. To address this challenging task, existing leading methods all resort to density map regression, which renders them impractical for downstream tasks that require object locations and restricts their ability to well explore the scale information of exemplars for supervision. To address the limitations, we propose a novel localization-based CAC approach, termed Scale-modulated Query and Localization Network (SQLNet). It fully explores the scales of exemplars in both the query and localization stages and achieves effective counting by accurately locating each object and predicting its approximate size. Specifically, during the query stage, rich discriminative representations of the target class are acquired by the Hierarchical Exemplars Collaborative Enhancement (HECE) module from the few exemplars through multi-scale exemplar cooperation with equifrequent size prompt embedding. These representations are then fed into the Exemplars-Unified Query Correlation (EUQC) module to interact with the query features in a unified manner and produce the correlated query tensor. In the localization stage, the Scale-aware Multi-head Localization (SAML) module utilizes the query tensor to predict the confidence, location, and size of each potential object. Moreover, a scale-aware localization loss is introduced, which exploits flexible location associations and exemplar scales for supervision to optimize the model performance. Extensive experiments demonstrate that SQLNet outperforms state-of-the-art methods on popular CAC benchmarks, achieving excellent performance not only in counting accuracy but also in localization and bounding box generation. Our codes will be available at https://github.com/HCPLab-SYSU/SQLNet
BibTeX
@article{wu2025sqlnet,
  author = {Hefeng Wu and Yandong Chen and Lingbo Liu and Tianshui Chen and Keze Wang and Liang Lin},
  title = {{SQLNet: Scale-Modulated Query and Localization Network for Few-Shot Class-Agnostic Counting}},
  journal = {IEEE Trans. Image Process.},
  year = {2025},
  doi = {10.1109/tip.2025.3588255},
}
Figure