TL;DR
When we evaluate text-to-image (T2I) models, we often treat the "verifier"—the model that scores the images—as a mere implementation detail. This paper demonstrates that verifier choice is actually a critical design variable that can radically distort benchmark results, even reversing the human-judged rankings of top-tier models like FLUX.2-dev and Qwen-Image.
What Problem Does This Paper Address?
Automatic evaluation is the backbone of modern T2I benchmarking. Typically, a researcher generates images from a model and uses a Multimodal Large Language Model (MLLM) or an object detector to check if the image matches the prompt. However, the community often ignores the fact that a benchmark score isn't just a property of the image generator; it’s a property of the entire pipeline, including the verifier model, the prompting protocol (e.g., zero-shot vs. few-shot), and the scoring rules.
The authors argue that treating verifiers as "black boxes" is unsafe. To prove this, they focus on structural counting—tasks like "generate a dresser with six drawers" or "a clover with four leaflets." Counting is an ideal "audit" task because the ground truth is visually unambiguous to humans, making it easy to spot when an automatic verifier hallucinates or fails.
Methods
The researchers constructed a verifier-centric audit suite. They generated 6,240 images using state-of-the-art models (Qwen-Image-2512, FLUX.2-dev, and Z-Image) and had humans manually audit them to establish a "gold standard" for counts and image validity.
They then tested a variety of verifiers against this human baseline:
- Open MLLMs: Qwen3.5 (4B, 9B, 27B) and Gemma (3n, 4).
- Frontier Closed Models: GPT-5.5 and Claude Opus 4.7.
- Specialist Detectors: A support-trained YOLO11s model.
Crucially, they didn't just test the models; they tested different protocols:
- Count-only: The verifier must provide a number.
- Reject-aware: The verifier can first decide if the image is "valid" (e.g., not blurry or malformed) before counting.
- Zero-shot vs. Few-shot: Providing examples in the prompt to guide the verifier.
Datasets
The paper introduces a new, human-audited structural-counting audit suite covering seven objects:
- Object Counting: Cell phones, chairs, and traffic lights.
- Substructure Counting: Clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades.
The evaluation used a balanced 2,280-image split to ensure statistically significant comparisons across different verifier configurations.
Key Insights and Ideas
- Aggregate scores are deceptive. A verifier might achieve an average score very close to a human's, but this can hide massive "image-level disagreement." In other words, the verifier might be right for the wrong reasons, passing images humans rejected and failing images humans passed.
- Verifiers "shrink" the gap between generators. Every verifier configuration tested reduced the measured performance difference between Qwen and FLUX compared to human judgments (by 7.5 to 25.0 percentage points). This suggests that automatic benchmarks may be making models look more similar than they actually are.
- Ranking Reversals. Most alarmingly, when comparing a third model (Z-Image) to FLUX.2-dev, all four tested automatic protocols reversed the human ordering. Humans preferred Qwen > Z-Image > FLUX, but the verifiers often put FLUX on top.
- Frontier models aren't perfect. Even the most powerful models like GPT-5.5 and Claude Opus 4.7 exhibited significant failure modes, such as low fidelity in counting clover leaflets or poor agreement on what constitutes a "valid" image.
Key Results
- Protocol Sensitivity: Moving from zero-shot to few-shot prompting changed the reliability (Cohen’s $\kappa$) significantly, but not always for the better. For some models, adding examples actually hurt performance.
- Directional Bias: Errors weren't random. For certain objects like traffic lights, verifiers showed a strong "over-counting" bias, while for substructures like turbine blades, they tended to "under-count."
- The "Reliability Profile": The authors propose that instead of a single number, benchmarks should report a "factorized verifier reliability profile" including metrics like image-level agreement, generator-comparison distortion, and rejection behavior.
Limitations and Future Work
The study is intentionally narrow, focusing on structural counting because it is highly auditable. The authors acknowledge that their findings might differ for more subjective tasks like "artistic style" or "overall quality."
Future work includes expanding this auditing framework to more complex compositional prompts and developing standardized "verifier selection cards" to help researchers choose the most reliable evaluation tools for their specific tasks. The main takeaway remains: an automatic score is not a ground truth; it is a measurement taken through a specific, and often biased, lens.
Summary updated