What Factors Matter for Collection-Level Counting with Human-AI Collaboration?

Seoyun Jeong, Logan Lawrence, Owen L. Martin, Benjamin Koger, Daniel Sheldon, Subhransu MajiarXiv 2026 · arXiv

TL;DR

This paper provides a comprehensive guide for "collection-level counting"—estimating the total number of objects across thousands of images—by combining AI predictions with strategic human verification. The authors demonstrate that using modern multimodal models to prioritize which images humans should label can reduce counting errors by 4x compared to traditional sampling, achieving high-precision results (±5% error) while requiring humans to review as little as 3.2% of a dataset.

What Problem Does This Paper Address?

In fields like wildlife conservation, disaster response, and medical pathology, researchers often need to know the total count of objects (e.g., migrating birds, damaged buildings, or infected cells) across massive image collections.

While AI models can count objects, they are often biased or struggle when deployed in new environments (domain shift). Conversely, having humans label every single image is prohibitively expensive and slow. The middle ground is human-AI collaboration: using a model to assist a statistical estimator that only requires a small fraction of human-verified labels to produce an unbiased total.

However, practitioners have lacked clear guidance on the best "recipe" for this: Should they use a pre-trained "foundation" model or fine-tune a custom detector? Should they sample images randomly or focus on where the model sees the most objects? This paper systematically evaluates these choices to provide a practical roadmap.

Methods

The authors utilize an Active Measurement framework, a sequential loop where:

  1. Prediction: An AI model predicts counts for every image in the collection.
  2. Sampling: A "sampler" selects an image for a human to verify.
  3. Estimation: An "estimator" combines the human labels with AI predictions to update the total count and confidence intervals.
  4. Adaptation (Optional): The human labels are used to fine-tune the AI model, improving future predictions.

The study compares several components:

  • Models: Fine-tuned detectors (Faster R-CNN), Open-Vocabulary Detectors (OVDs like Florence-2), and Multimodal Large Language Models (MLLMs like InternVL3.5 and GPT-5.2).
  • Samplers: Uniform sampling vs. Importance Sampling (prioritizing images where the model predicts high counts).
  • Estimators: Classical Monte Carlo vs. model-assisted variants (Difference, Ratio, and Regression estimators).

Datasets

The researchers introduced a new benchmark consisting of 12 counting tasks derived from four public datasets:

  • Malaria Cell Images: Counting different types of infected blood cells.
  • Birds: Counting swallows in ground-level flock photos.
  • Kenyan Wildlife Aerial Survey: Counting giraffes from the air.
  • DOTA v1.5: Counting urban objects (planes, ships, storage tanks) in satellite imagery.

These tasks are categorized into "occupancy tiers" based on how "rare" the objects are (from <5% empty images to >90% empty), which is a critical factor in determining the best counting strategy.

Figure 1: Samples from the 12 counting tasks. Each in-image label identifies the task (dataset/class), tile index, and exact ground-truth count. Red points mark each object.

Key Insights and Ideas

  • MLLMs are the new "Gold Standard" for starting out: Open-source MLLMs (like InternVL3.5-8B) are remarkably strong "zero-shot" counters. They outperform older OVDs and provide a high-quality starting point without requiring any initial training data.
  • Importance Sampling is a "Must-Have": If your model can at least rank images correctly (even if its absolute counts are wrong), using Importance Sampling (DIS/AIS) is significantly more efficient than uniform sampling, especially for rare objects.
  • The "Crossover" Point: While MLLMs win at low labeling budgets (under 30% of the collection), custom fine-tuned detectors eventually overtake them as more human labels become available for training.
  • Pilot Sets for Model Selection: You don't need to guess which model will work best. The authors found that labeling a tiny "pilot set" of 50 random images and calculating the Hellinger distance between the model's predicted distribution and the human counts is a highly reliable way to pick the best model for the full task.
Overview of results showing that model-assisted estimators (below the 1x line) consistently beat model-free baselines across different object densities.

Key Results

  • Efficiency Gains: On the benchmark, the best human-AI strategies reached under 10% error with less than 1% of the collection labeled—a 4x improvement over human-only sampling.
  • Model Performance: InternVL3.5-8B was the most frequent winner among frozen models, particularly at low budgets.
  • Real-World Impact: In a case study on aerial pronghorn surveys in Wyoming, the team used these methods to meet a strict ±5% uncertainty tolerance after reviewing a median of only 3.2% of the images. This effectively replaces a dangerous manual task (flying low-altitude observers) with a highly efficient, verifiable digital workflow.
Error curves for the Birds task showing how different estimators converge as more data is labeled.

Limitations and Future Work

  • Single-Class Focus: The current study focuses on counting one type of object at a time. Real-world surveys often require multi-class counting (e.g., elephants, zebras, and giraffes simultaneously), which adds complexity to sampling strategies.
  • Cost and Latency: While open-source MLLMs are powerful, they are computationally expensive to run over millions of images compared to lightweight detectors.
  • Human Error: The framework assumes human labels are "ground truth." In practice, humans also make mistakes, especially in crowded or blurry images.
  • Future Directions: The authors suggest exploring "distillation"—using expensive MLLMs to train smaller, faster detectors—and developing more robust methods for multi-class active measurement.

Summary updated

Abstract

Many scientific and policy decisions require estimating the total number of target objects across a large image collection. Exhaustive human labeling is often infeasible, while raw model predictions can be biased, especially under domain shift. Recent human-AI estimation methods combine imperfect model predictions with limited human verification to obtain reliable collection-level estimates. However, existing work provides little guidance on how practitioners should choose among models, estimators, and adaptation strategies for a new deployment. We study these choices within a common human-in-the-loop counting framework across 12 tasks spanning biomedical microscopy, ground-level and aerial wildlife surveys, and overhead imagery of urban structures. Our results show that modern multimodal models provide a strong, training-free starting point for many tasks, while fine-tuned detectors become more advantageous at larger labeling budgets and across repeated deployments. Estimation efficiency improves further when model predictions are used to prioritize which units humans label, rather than sampling uniformly. We also find that each model's ranking quality can be estimated with a small pilot set, yielding practical guidance for selecting strategies on new counting tasks. In a field deployment on aerial pronghorn surveys, a task-specific adaptation of these methods meets the survey's $\pm 5\%$ tolerance after reviewing a median of only $3.2\%$ of collected images.
BibTeX
@misc{jeong2026what,
  author = {Seoyun Jeong and Logan Lawrence and Owen L. Martin and Benjamin Koger and Daniel Sheldon and Subhransu Maji},
  title = {{What Factors Matter for Collection-Level Counting with Human-AI Collaboration?}},
  year = {2026},
  eprint = {2611.14380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14380},
}
Figure