Mind the Prompt: A Novel Benchmark for Prompt-Based Class-Agnostic Counting

Luca Ciampi, Nicola Messina, Matteo Pierucci, Giuseppe Amato, Marco Avvenuti, Fabrizio FalchiWACV 2025

TL;DR

This paper introduces PrACo, a new benchmark designed to rigorously evaluate prompt-based class-agnostic counting (CAC) models. It highlights significant limitations in current evaluation methods that mask a fundamental weakness in these models: their inability to truly understand textual prompts and distinguish between target and distractor objects, even when achieving high scores on traditional metrics.

What Problem Does This Paper Address?

Class-agnostic counting (CAC) aims to count objects of arbitrary classes, even those unseen during training. A promising approach is prompt-based CAC, where users specify the object class using natural language. However, existing benchmarks for this task are flawed. They often use datasets with single-class images, making it difficult to assess a model's ability to differentiate between multiple object types. Furthermore, current evaluation metrics, inherited from class-specific counting, focus solely on counting errors and ignore whether the model actually understood the requested object category. This leaves a critical gap in evaluating the robustness and trustworthiness of these models, potentially leading to misinterpretations in real-world applications.

Methods

The authors introduce PrACo, a novel benchmark comprising two tailored test suites:

  • Negative-label Test: This test evaluates single-class images by prompting the model with descriptions of absent object classes. The goal is to see if the model incorrectly counts objects that are not present.
  • Mosaic Test: This test uses artificially created images by stitching together positive (target object) and negative (distractor object) single-class images. The model is prompted with the target class, and the test assesses its ability to count the target objects while ignoring the distractor objects.

To quantitatively assess these tests, PrACo introduces two new metrics:

  • Normalized Mean of Negative predictions (NMN): Measures the average absolute counting error when prompted with negative classes, normalized by the ground truth of the positive class. Lower NMN is better.
  • Positive Class Count Nearness (PCCN): Measures the percentage of samples where the model's positive class count estimate is closer to the ground truth than the mean of its negative class estimates. Higher PCCN is better, indicating the model is less biased towards counting negative classes.

Additionally, for the mosaic test, they define:

  • Counting Precision (CntP): Measures how well the model avoids counting distractor objects.
  • Counting Recall (CntR): Measures how well the model counts the target objects.
  • Counting Balanced F1-score (CntF1): The harmonic mean of CntP and CntR, providing an overall performance measure.

Datasets

The PrACo benchmark utilizes the FSC-147 dataset for its evaluations. FSC-147 contains images across 147 object categories with annotations as dots over object centroids. The authors filter out multi-class images from FSC-147 to ensure single-class images for the negative-label test and use pairs of these single-class images to construct the mosaic test images.

Key Insights and Ideas

  • The "Prompt Blindness" Phenomenon: State-of-the-art prompt-based CAC models, despite achieving impressive results on standard metrics like MAE and RMSE, often fail to truly understand the textual prompt. They can be easily distracted by objects of other classes or even count non-existent objects when prompted.
  • Limitations of Existing Benchmarks: The paper strongly argues that current datasets (often single-class) and metrics (focused solely on count error) are insufficient for evaluating the nuanced understanding required for prompt-based CAC.
  • Two-Stage Models Show Promise but Have Weaknesses: While two-stage models (like DAVE and TFPOC) generally perform better on the negative-label test, they can still exhibit catastrophic failures in the mosaic test due to their verification modules.
  • Underestimated Importance of Negative Examples: The study highlights that handling negative examples (distractors) is a significant challenge that is often overlooked in current evaluations.

Key Results

  • On standard metrics (MAE, RMSE), several methods achieve comparable, high performance. However, on PrACo's novel metrics, the performance is "quite heterogeneous."
  • DAVE (a two-stage model) shows the best performance on the negative test (lowest NMN, highest PCCN) and strong results on the mosaic test (highest CntP and CntF1).
  • TFPOC (also a two-stage model) shows competitive results on PrACo metrics, even outperforming some learning-based models, suggesting its training-free approach might be more robust to prompt ambiguity.
  • CounTX, CLIP-Count, and VLCounter (one-stage models) exhibit significant weaknesses on the negative test, with NMN values close to or above 1, indicating their negative class predictions are comparable to the ground truth of the positive class.
  • Qualitative results (Figure 5) show that while models can often estimate the positive class count correctly in mosaics, they frequently misidentify or count instances from the negative image.

Limitations and Future Work

  • Dataset Limitations: While FSC-147 is used, the paper notes that real-world scenarios might involve more complex multi-class images than those artificially constructed for the mosaic test.
  • Model Weaknesses: The paper explicitly points out that current SOTA models need better training procedures and architectural designs to improve their understanding of textual prompts and handle negative examples more effectively.
  • Future Directions: The authors hope PrACo will serve as a foundation for developing more robust prompt-based CAC methods and encourage more comprehensive evaluations that go beyond simple counting error. They suggest further research into how models interpret prompts and handle ambiguity.

Summary updated

BibTeX
@inproceedings{ciampi2025mind,
  author = {Luca Ciampi and Nicola Messina and Matteo Pierucci and Giuseppe Amato and Marco Avvenuti and Fabrizio Falchi},
  title = {{Mind the Prompt: A Novel Benchmark for Prompt-Based Class-Agnostic Counting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure