TL;DR
This paper introduces PROBEGEN, a rigorous benchmark evaluating whether general-purpose image generators can perform visual perception tasks out-of-the-box via text prompting. The study reveals a fundamental trade-off: while discriminative specialists dominate in speed and in-distribution accuracy, image generators exhibit remarkable robustness under domain shifts (like art and manga) and excel at complex, compositional semantic reasoning.
What Problem Does This Paper Address?
State-of-the-art image generators must internalize a deep understanding of geometry, object structures, and spatial relationships to synthesize realistic images. This raises an intriguing question: Can these internal representations be directly exploited for visual perception tasks without any task-specific fine-tuning or architectural modifications?
While prior works have adapted diffusion models into perception tools through fine-tuning, this paper tackles the unexplored regime of zero-shot generative perception. The authors aim to establish a controlled, head-to-head comparison between mainstream image generators, discriminative specialist models (like Segment Anything and Depth Anything), and Multimodal Large Language Models (MLLMs) across standard public benchmarks.
Methods
The authors formalize zero-shot generative perception by casting tasks as conditional image generation problems specified entirely through text prompts. To evaluate the models fairly, they introduce a unified inference protocol that decouples a model's raw output from the final metric evaluation using fixed, deterministic extractors:
- Segmentation: Prompts request a grayscale mask (white foreground, black background). The output is binarized using Otsu’s adaptive thresholding to handle soft-boundary artifacts.
- Monocular Depth Estimation: Prompts request a grayscale depth map (brighter = closer). Predictions are aligned to the ground truth using an affine-invariant scale-and-shift fitting protocol to resolve linear ambiguities.
- Object Counting: Models are evaluated via a "mark-then-count" interface, where they are prompted to paint a small red dot on every target object. A three-stage deterministic decoder isolates the color, filters out pre-existing red structures using a difference gate, and counts the dots via a shape-filtered blob detector.
The protocol also evaluates prompt sensitivity across different granularities (short, mid, and long prompt templates).
Datasets
The evaluation spans 11 published benchmarks across three core perception tasks, utilizing a randomly sampled subset of 500 images per benchmark (5,500 total samples):
- Monocular Depth: NYUv2, DIODE, ScanNet, and KITTI.
- Referring/Reasoning Segmentation: RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg.
- Out-of-Distribution (OOD) Segmentation: Manga109 (manga imagery) and DRAM (art paintings).
- Object Counting: PixMo-Points.
Key Insights and Ideas
- The Accuracy-Robustness Trade-off: Specialists win on standard, in-distribution natural images. However, when tested on out-of-distribution (OOD) domains like stylized art or manga, specialists frequently over- or under-segment. Generative models leverage their broad visual pretraining priors to maintain semantic grounding where specialists fail.
- Semantic Grounding vs. Spatial Faithfulness: Generative generalists excel at understanding what to look for—even succeeding on complex, multi-step reasoning queries. However, they struggle with spatial faithfulness, often failing to modify the input image strictly in place.
- Generative Failure Modes: The authors categorize two distinct failure modes unique to generators: input-preservation failure (where the model re-renders or hallucinates new background content instead of transforming the image in place) and instruction-following failure (where the model returns a stylized copy rather than a structured mask or depth map).
Key Results
- Reasoning & OOD Segmentation: On the complex
ReasonSegbenchmark, Nano Banana 2 achieved the highest score among directly comparable systems with 70.3 gIoU, outperforming a multi-step agentic pipeline combining SAM3 and Qwen3-VL (62.5 gIoU). - OOD Superiority: On the artistic
DRAMdataset, Nano Banana 2 reached 82.5 gIoU, significantly outperforming the specialist SAM3 baseline (63.0 gIoU), as detailed in the quantitative breakdown:
- Qualitative Competence: Qualitative examples highlight how a single generative model can successfully output relative depth maps, count objects, and segment complex queries zero-shot:
- Object Counting: Text-based MLLMs (like Gemini 3.5 Flash) lead in Mean Absolute Error (MAE), but image generators demonstrate superior worst-case robustness (lower RMSE) in densely populated scenes because their mark-then-count approach degrades more gracefully than direct numerical estimation.
Limitations and Future Work
- Computational Efficiency: Generative models are currently 1 to 3 orders of magnitude slower than discriminative specialists and demand substantially higher GPU memory, making them impractical for real-time deployment.
- Unsolved Tasks: Zero-shot surface normal estimation remains a challenge; tested generators produced visually plausible maps that were metrically highly inaccurate.
- Future Extensions: The authors propose scaling the benchmark to full datasets, incorporating multi-sample inference techniques (like self-consistency voting), and expanding the output interfaces to include tasks like bounding-box drawing for object detection and keypoint rendering for human pose estimation.
Summary updated