TL;DR
Open Vision Banana (OVB) is a fully open-source framework that transforms pre-trained image generators into powerful multi-task perception systems by treating vision tasks as image-to-image generation. By fine-tuning open-weight models like FLUX.2-Klein, OVB achieves state-of-the-art zero-shot performance on benchmarks like Cityscapes and introduces "legend ensembling"—a training-free method to boost accuracy by up to 15 points at test time.
What Problem Does This Paper Address?
Recent research has shown that large-scale image generators (like Stable Diffusion or FLUX) develop a deep latent understanding of the physical world to create realistic images. Models like the proprietary "Vision Banana" have successfully tapped into this by framing perception tasks—such as segmentation and depth estimation—as conditional image editing.
However, the "recipe" for Vision Banana (its data, weights, and specific training choices) remains a secret. This lack of transparency prevents the research community from reproducing results, understanding which design choices actually matter, or building upon the work. OVB fills this gap by providing a completely open, reproducible pipeline for turning open-weight generators into generalist vision models.
Methods
The authors adapt the FLUX.2-Klein (4B and 9B parameter) models. Instead of adding task-specific "heads" (the standard approach in computer vision), they treat every task as an image-to-image problem.
Key technical components include:
- RGB Task Modeling: Perception outputs are rendered as RGB images. For example, semantic segmentation maps use specific colors for different classes, and depth is represented as a grayscale "inverse-depth" map.
- Prompt-Driven Perception: The model receives an input image and a text prompt (e.g., "Segment the road in blue and the car in red").
- Legend Ensembling: This is OVB's "secret sauce" for test-time scaling. Because a model might occasionally "hallucinate" the wrong color for a class, OVB runs the same image through the model multiple times with different class-color mappings (legends). It then uses a pixel-wise majority vote to fuse these results, significantly correcting errors without any additional training.
- Full Transformer Tuning: Unlike methods that only tune small "adapter" layers (LoRA), OVB finds that updating all transformer parameters is crucial for high-quality segmentation and normal estimation.
Datasets
OVB is trained exclusively on public datasets, using a mixture of 2.1M to 3.8M training pairs.
- Segmentation: ACDC, BDD100K, ADE20K, PASCAL VOC, Mapillary Vistas, COCO-Stuff, and more.
- Geometry: Synscapes, Replica, Matterport3D, Hypersim, and Taskonomy.
- General Capabilities: To ensure the model doesn't lose its ability to generate and edit images, they include "replay" data from Pico-Banana-400K (editing) and PD12M (text-to-image).
- Evaluation: Benchmarked on Cityscapes, COCO-Object, RefCOCOg (referring segmentation), ReasonSeg (reasoning segmentation), and NYUv2 (depth/normals).
Key Insights and Ideas
- Randomization is Key: Training with fixed class-color mappings (e.g., always "car=blue") leads to rigid memorization. Using randomized colors in prompts forces the model to actually "read" the legend and follow instructions, leading to better generalization.
- The "Top 64" Sweet Spot: When choosing a color palette for segmentation, there's a trade-off. Too many colors lead to ambiguity; too few limit the number of classes. The authors found that a palette of 64 well-separated colors provides the best overall performance.
- Task Interaction: Design choices are not independent. A choice that helps one task might hurt another. For instance, while freezing embeddings helps depth accuracy, it significantly degrades semantic segmentation.
- Generative Priors are Robust: Even without seeing a specific dataset during training (zero-shot), the 9B model's internal representation is strong enough to outperform many specialized models on complex urban scenes like Cityscapes.
Key Results
- Cityscapes Zero-Shot: OVB 9B achieved 73.9 mIoU, surpassing the proprietary Vision Banana (69.9 mIoU) and the zero-shot specialist SAM (65.2 mIoU).
- Legend Ensembling Gains: On COCO-Object, legend ensembling improved mIoU by 15.0 points simply by aggregating results from multiple passes.
- Geometry: OVB rivals state-of-the-art affine-invariant depth estimators, showing that generative models are excellent at capturing spatial layout.
- Feature Quality: Fine-tuning for perception actually improves the model's internal features for downstream tasks like unsupervised object discovery, even surpassing DINOv3 in some metrics.
Limitations and Future Work
Despite its strengths, OVB still trails proprietary models in instance segmentation (identifying individual objects of the same class). The authors note that the perception ability of current open-weight base models (like FLUX.2-Klein) is still fundamentally weaker than top-tier proprietary generators like "Nano Banana Pro."
Future work could involve scaling to even larger base models, improving the "readout" logic for instance segmentation, and exploring how to further reduce the computational cost of legend ensembling at inference time.
Summary updated