Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation

Shangzhe Di, Zhaokai Wang, Weidi XiearXiv 2026 · arXiv

TL;DR

This paper introduces PROBEGEN, a rigorous benchmark evaluating whether general-purpose image generators can perform visual perception tasks out-of-the-box via text prompting. The study reveals a fundamental trade-off: while discriminative specialists dominate in speed and in-distribution accuracy, image generators exhibit remarkable robustness under domain shifts (like art and manga) and excel at complex, compositional semantic reasoning.

What Problem Does This Paper Address?

State-of-the-art image generators must internalize a deep understanding of geometry, object structures, and spatial relationships to synthesize realistic images. This raises an intriguing question: Can these internal representations be directly exploited for visual perception tasks without any task-specific fine-tuning or architectural modifications?

While prior works have adapted diffusion models into perception tools through fine-tuning, this paper tackles the unexplored regime of zero-shot generative perception. The authors aim to establish a controlled, head-to-head comparison between mainstream image generators, discriminative specialist models (like Segment Anything and Depth Anything), and Multimodal Large Language Models (MLLMs) across standard public benchmarks.

Methods

The authors formalize zero-shot generative perception by casting tasks as conditional image generation problems specified entirely through text prompts. To evaluate the models fairly, they introduce a unified inference protocol that decouples a model's raw output from the final metric evaluation using fixed, deterministic extractors:

  • Segmentation: Prompts request a grayscale mask (white foreground, black background). The output is binarized using Otsu’s adaptive thresholding to handle soft-boundary artifacts.
  • Monocular Depth Estimation: Prompts request a grayscale depth map (brighter = closer). Predictions are aligned to the ground truth using an affine-invariant scale-and-shift fitting protocol to resolve linear ambiguities.
  • Object Counting: Models are evaluated via a "mark-then-count" interface, where they are prompted to paint a small red dot on every target object. A three-stage deterministic decoder isolates the color, filters out pre-existing red structures using a difference gate, and counts the dots via a shape-filtered blob detector.

The protocol also evaluates prompt sensitivity across different granularities (short, mid, and long prompt templates).

Datasets

The evaluation spans 11 published benchmarks across three core perception tasks, utilizing a randomly sampled subset of 500 images per benchmark (5,500 total samples):

  • Monocular Depth: NYUv2, DIODE, ScanNet, and KITTI.
  • Referring/Reasoning Segmentation: RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg.
  • Out-of-Distribution (OOD) Segmentation: Manga109 (manga imagery) and DRAM (art paintings).
  • Object Counting: PixMo-Points.

Key Insights and Ideas

  • The Accuracy-Robustness Trade-off: Specialists win on standard, in-distribution natural images. However, when tested on out-of-distribution (OOD) domains like stylized art or manga, specialists frequently over- or under-segment. Generative models leverage their broad visual pretraining priors to maintain semantic grounding where specialists fail.
  • Semantic Grounding vs. Spatial Faithfulness: Generative generalists excel at understanding what to look for—even succeeding on complex, multi-step reasoning queries. However, they struggle with spatial faithfulness, often failing to modify the input image strictly in place.
  • Generative Failure Modes: The authors categorize two distinct failure modes unique to generators: input-preservation failure (where the model re-renders or hallucinates new background content instead of transforming the image in place) and instruction-following failure (where the model returns a stylized copy rather than a structured mask or depth map).

Key Results

  • Reasoning & OOD Segmentation: On the complex ReasonSeg benchmark, Nano Banana 2 achieved the highest score among directly comparable systems with 70.3 gIoU, outperforming a multi-step agentic pipeline combining SAM3 and Qwen3-VL (62.5 gIoU).
  • OOD Superiority: On the artistic DRAM dataset, Nano Banana 2 reached 82.5 gIoU, significantly outperforming the specialist SAM3 baseline (63.0 gIoU), as detailed in the quantitative breakdown:
Reasoning and OOD Segmentation Performance
  • Qualitative Competence: Qualitative examples highlight how a single generative model can successfully output relative depth maps, count objects, and segment complex queries zero-shot:
Zero-shot perception abilities of Nano Banana 2
  • Object Counting: Text-based MLLMs (like Gemini 3.5 Flash) lead in Mean Absolute Error (MAE), but image generators demonstrate superior worst-case robustness (lower RMSE) in densely populated scenes because their mark-then-count approach degrades more gracefully than direct numerical estimation.

Limitations and Future Work

  • Computational Efficiency: Generative models are currently 1 to 3 orders of magnitude slower than discriminative specialists and demand substantially higher GPU memory, making them impractical for real-time deployment.
  • Unsolved Tasks: Zero-shot surface normal estimation remains a challenge; tested generators produced visually plausible maps that were metrically highly inaccurate.
  • Future Extensions: The authors propose scaling the benchmark to full datasets, incorporating multi-sample inference techniques (like self-consistency voting), and expanding the output interfaces to include tasks like bounding-box drawing for object detection and keypoint rendering for human pose estimation.

Summary updated

Abstract

Recent work, such as Vision Banana, shows that lightweight instruction tuning can enable an image generator to achieve state-of-the-art performance across multiple visual perception tasks. Motivated by this perspective, we ask how far image generators can go on public visual perception benchmarks in a zero-shot setting. We introduce ProbeGen, a benchmark for zero-shot generative perception that casts monocular depth estimation, referring/reasoning segmentation, and object counting as conditional generation tasks specified through text prompts, and compares 20 models in total---including proprietary and open-weight image generators, specialist perception models, and MLLMs---across 11 published benchmarks. We observe that pretrained image generators show measurable zero-shot perceptual competence, but with a clear trade-off: specialist models remain stronger for in-distribution accuracy and efficiency, while generative models are often more robust under distribution shift and better at compositional semantic reasoning. We hope this study helps establish zero-shot generative perception as a meaningful research direction and provides a useful foundation for future work at the intersection of visual generation and understanding.
BibTeX
@misc{di2026are,
  author = {Shangzhe Di and Zhaokai Wang and Weidi Xie},
  title = {{Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation}},
  year = {2026},
  eprint = {2609.07884},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.07884},
}
Figure