TL;DR
This paper provides a comprehensive guide for "collection-level counting"—estimating the total number of objects across thousands of images—by combining AI predictions with strategic human verification. The authors demonstrate that using modern multimodal models to prioritize which images humans should label can reduce counting errors by 4x compared to traditional sampling, achieving high-precision results (±5% error) while requiring humans to review as little as 3.2% of a dataset.
What Problem Does This Paper Address?
In fields like wildlife conservation, disaster response, and medical pathology, researchers often need to know the total count of objects (e.g., migrating birds, damaged buildings, or infected cells) across massive image collections.
While AI models can count objects, they are often biased or struggle when deployed in new environments (domain shift). Conversely, having humans label every single image is prohibitively expensive and slow. The middle ground is human-AI collaboration: using a model to assist a statistical estimator that only requires a small fraction of human-verified labels to produce an unbiased total.
However, practitioners have lacked clear guidance on the best "recipe" for this: Should they use a pre-trained "foundation" model or fine-tune a custom detector? Should they sample images randomly or focus on where the model sees the most objects? This paper systematically evaluates these choices to provide a practical roadmap.
Methods
The authors utilize an Active Measurement framework, a sequential loop where:
- Prediction: An AI model predicts counts for every image in the collection.
- Sampling: A "sampler" selects an image for a human to verify.
- Estimation: An "estimator" combines the human labels with AI predictions to update the total count and confidence intervals.
- Adaptation (Optional): The human labels are used to fine-tune the AI model, improving future predictions.
The study compares several components:
- Models: Fine-tuned detectors (Faster R-CNN), Open-Vocabulary Detectors (OVDs like Florence-2), and Multimodal Large Language Models (MLLMs like InternVL3.5 and GPT-5.2).
- Samplers: Uniform sampling vs. Importance Sampling (prioritizing images where the model predicts high counts).
- Estimators: Classical Monte Carlo vs. model-assisted variants (Difference, Ratio, and Regression estimators).
Datasets
The researchers introduced a new benchmark consisting of 12 counting tasks derived from four public datasets:
- Malaria Cell Images: Counting different types of infected blood cells.
- Birds: Counting swallows in ground-level flock photos.
- Kenyan Wildlife Aerial Survey: Counting giraffes from the air.
- DOTA v1.5: Counting urban objects (planes, ships, storage tanks) in satellite imagery.
These tasks are categorized into "occupancy tiers" based on how "rare" the objects are (from <5% empty images to >90% empty), which is a critical factor in determining the best counting strategy.
Key Insights and Ideas
- MLLMs are the new "Gold Standard" for starting out: Open-source MLLMs (like InternVL3.5-8B) are remarkably strong "zero-shot" counters. They outperform older OVDs and provide a high-quality starting point without requiring any initial training data.
- Importance Sampling is a "Must-Have": If your model can at least rank images correctly (even if its absolute counts are wrong), using Importance Sampling (DIS/AIS) is significantly more efficient than uniform sampling, especially for rare objects.
- The "Crossover" Point: While MLLMs win at low labeling budgets (under 30% of the collection), custom fine-tuned detectors eventually overtake them as more human labels become available for training.
- Pilot Sets for Model Selection: You don't need to guess which model will work best. The authors found that labeling a tiny "pilot set" of 50 random images and calculating the Hellinger distance between the model's predicted distribution and the human counts is a highly reliable way to pick the best model for the full task.
Key Results
- Efficiency Gains: On the benchmark, the best human-AI strategies reached under 10% error with less than 1% of the collection labeled—a 4x improvement over human-only sampling.
- Model Performance: InternVL3.5-8B was the most frequent winner among frozen models, particularly at low budgets.
- Real-World Impact: In a case study on aerial pronghorn surveys in Wyoming, the team used these methods to meet a strict ±5% uncertainty tolerance after reviewing a median of only 3.2% of the images. This effectively replaces a dangerous manual task (flying low-altitude observers) with a highly efficient, verifiable digital workflow.
Limitations and Future Work
- Single-Class Focus: The current study focuses on counting one type of object at a time. Real-world surveys often require multi-class counting (e.g., elephants, zebras, and giraffes simultaneously), which adds complexity to sampling strategies.
- Cost and Latency: While open-source MLLMs are powerful, they are computationally expensive to run over millions of images compared to lightweight detectors.
- Human Error: The framework assumes human labels are "ground truth." In practice, humans also make mistakes, especially in crowded or blurry images.
- Future Directions: The authors suggest exploring "distillation"—using expensive MLLMs to train smaller, faster detectors—and developing more robust methods for multi-class active measurement.
Summary updated