Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models

Shurun Li, Michael Hua Wang, Haibo Zeng, Long WangarXiv 2026 · arXiv

TL;DR

When we evaluate text-to-image (T2I) models, we often treat the "verifier"—the model that scores the images—as a mere implementation detail. This paper demonstrates that verifier choice is actually a critical design variable that can radically distort benchmark results, even reversing the human-judged rankings of top-tier models like FLUX.2-dev and Qwen-Image.

What Problem Does This Paper Address?

Automatic evaluation is the backbone of modern T2I benchmarking. Typically, a researcher generates images from a model and uses a Multimodal Large Language Model (MLLM) or an object detector to check if the image matches the prompt. However, the community often ignores the fact that a benchmark score isn't just a property of the image generator; it’s a property of the entire pipeline, including the verifier model, the prompting protocol (e.g., zero-shot vs. few-shot), and the scoring rules.

The authors argue that treating verifiers as "black boxes" is unsafe. To prove this, they focus on structural counting—tasks like "generate a dresser with six drawers" or "a clover with four leaflets." Counting is an ideal "audit" task because the ground truth is visually unambiguous to humans, making it easy to spot when an automatic verifier hallucinates or fails.

Figure 1: Different verifiers and protocols can produce wildly different scores for the same generated image, illustrating why verifier choice must be treated as a design variable.

Methods

The researchers constructed a verifier-centric audit suite. They generated 6,240 images using state-of-the-art models (Qwen-Image-2512, FLUX.2-dev, and Z-Image) and had humans manually audit them to establish a "gold standard" for counts and image validity.

They then tested a variety of verifiers against this human baseline:

  • Open MLLMs: Qwen3.5 (4B, 9B, 27B) and Gemma (3n, 4).
  • Frontier Closed Models: GPT-5.5 and Claude Opus 4.7.
  • Specialist Detectors: A support-trained YOLO11s model.

Crucially, they didn't just test the models; they tested different protocols:

  • Count-only: The verifier must provide a number.
  • Reject-aware: The verifier can first decide if the image is "valid" (e.g., not blurry or malformed) before counting.
  • Zero-shot vs. Few-shot: Providing examples in the prompt to guide the verifier.

Datasets

The paper introduces a new, human-audited structural-counting audit suite covering seven objects:

  1. Object Counting: Cell phones, chairs, and traffic lights.
  2. Substructure Counting: Clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades.

The evaluation used a balanced 2,280-image split to ensure statistically significant comparisons across different verifier configurations.

Key Insights and Ideas

  • Aggregate scores are deceptive. A verifier might achieve an average score very close to a human's, but this can hide massive "image-level disagreement." In other words, the verifier might be right for the wrong reasons, passing images humans rejected and failing images humans passed.
  • Verifiers "shrink" the gap between generators. Every verifier configuration tested reduced the measured performance difference between Qwen and FLUX compared to human judgments (by 7.5 to 25.0 percentage points). This suggests that automatic benchmarks may be making models look more similar than they actually are.
  • Ranking Reversals. Most alarmingly, when comparing a third model (Z-Image) to FLUX.2-dev, all four tested automatic protocols reversed the human ordering. Humans preferred Qwen > Z-Image > FLUX, but the verifiers often put FLUX on top.
  • Frontier models aren't perfect. Even the most powerful models like GPT-5.5 and Claude Opus 4.7 exhibited significant failure modes, such as low fidelity in counting clover leaflets or poor agreement on what constitutes a "valid" image.
Figure 2: Aggregate score gap and image-level agreement for the current verifier–protocol pairs. The shaded band marks ±2 pp around score parity as a visual reference. Error bars are marginal 95% image-bootstrap intervals.

Key Results

  • Protocol Sensitivity: Moving from zero-shot to few-shot prompting changed the reliability (Cohen’s $\kappa$) significantly, but not always for the better. For some models, adding examples actually hurt performance.
  • Directional Bias: Errors weren't random. For certain objects like traffic lights, verifiers showed a strong "over-counting" bias, while for substructures like turbine blades, they tended to "under-count."
  • The "Reliability Profile": The authors propose that instead of a single number, benchmarks should report a "factorized verifier reliability profile" including metrics like image-level agreement, generator-comparison distortion, and rejection behavior.

Limitations and Future Work

The study is intentionally narrow, focusing on structural counting because it is highly auditable. The authors acknowledge that their findings might differ for more subjective tasks like "artistic style" or "overall quality."

Future work includes expanding this auditing framework to more complex compositional prompts and developing standardized "verifier selection cards" to help researchers choose the most reliable evaluation tools for their specific tasks. The main takeaway remains: an automatic score is not a ground truth; it is a measurement taken through a specific, and often biased, lens.

Summary updated

Abstract

Automatic evaluation is now central to text-to-image benchmarking, but the verifier that converts generated images into scores is often treated as an implementation detail. We argue that this is unsafe: an automatic benchmark score is a property of the image generator, prompt distribution, verifier, verifier-use protocol, and scoring rule, not of the generator alone. We study this issue in structural counting, a controlled setting where human-visible counts can be audited. We construct a verifier-centric audit suite over seven counting objects: cell phones, chairs, and traffic lights for object counting, and clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades for substructure counting. Across 6,240 generated images from Qwen-Image-2512 and FLUX.2-dev, we compare open MLLM verifiers under count-only, reject-aware, zero-shot and few-shot protocols, plus TIFA-like target queries, a detector baseline, and frontier closed-model checks on two hard objects. On a balanced 2,280-image evaluation split, aggregate score matching can coexist with image-level disagreement, and verifier-use protocols and generator distributions change the measured reliability. Every verifier-protocol configuration also shrinks the human score difference between the two generators, by 7.5-25.0 pp, and with a third generator, Z-Image, all four tested protocols reverse the human ordering of Z-Image and FLUX.2-dev on our fixed prompt suite. GPT-5.5 and Claude Opus 4.7 checks do not remove the failure modes: GPT-5.5 still has only 59.92% clover valid-count fidelity, while Claude Opus 4.7 has lower same-slice agreement ($κ=0.432$). We propose reporting automatic benchmark scores with a factorized verifier reliability profile: aggregate score gap, image-level agreement, generator-comparison distortion, valid-image count fidelity, rejection behavior, output compliance, and directional bias.
BibTeX
@misc{li2026verifier,
  author = {Shurun Li and Michael Hua Wang and Haibo Zeng and Long Wang},
  title = {{Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models}},
  year = {2026},
  eprint = {2611.15433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15433},
}
Figure