TL;DR
Iris-Z demonstrates that a single generative model can outperform specialized architectures in monocular depth estimation through scale and joint RGB-D training. By treating depth as just another image modality within a unified transformer backbone, the authors show that "the bitter lesson"—that general methods scaled with compute eventually win—applies to spatial intelligence as much as it does to language.
What Problem Does This Paper Address?
In the world of computer vision, we have traditionally relied on a "fragmented" approach: one specialized model for segmentation, another for depth estimation, and yet another for image generation. While Large Language Models (LLMs) have successfully unified diverse text tasks into a single generative framework, visual intelligence has remained largely task-specific.
The authors of Iris-Z challenge this status quo. They address the question: Can a general-purpose image generator learn the underlying geometry of the world well enough to beat "specialist" models designed solely for depth estimation? Specifically, they target monocular geometry—the difficult task of estimating 3D depth from a single 2D photograph.
Methods
Iris-Z is built upon the FLUX.2 family of Diffusion Transformers (DiT). The core innovation is the move to a unified RGB-D sequence. Instead of adding a "depth head" to a vision model, the authors treat depth as an image.
Key technical highlights include:
- Shared Backbone: A single set of weights handles both appearance (RGB) and geometry (Depth). Both modalities are encoded into latents using the same VAE encoder.
- Modality Embeddings: The model uses learned embeddings to distinguish between generated RGB, generated depth, and "given" (conditioning) inputs.
- Joint Denoising: During training, the model learns to denoise image and depth latents together. Because the transformer's attention mechanism runs over the entire sequence, the image and depth tokens condition each other at every step.
- Multi-Task Training: The model is trained on a mix of tasks: generating RGB-D from a prompt, estimating depth from an image, and even "painting" an image onto a provided depth map.
Datasets
The models were trained on a massive pool of 3 to 4 million unique RGB-D frames. This data includes indoor captures, driving scenes with LiDAR, stereo reconstructions, and synthetic renders.
For evaluation, the authors tested Iris-Z against two major benchmark protocols:
- Marigold’s Protocol: NYUv2, KITTI, ETH3D, ScanNet, and DIODE.
- MoGe-3’s Protocol: NYUv2, KITTI, ETH3D, iBims-1, Sintel, DIODE, and HAMMER.
Crucially, the authors did not train on any of these benchmark datasets, ensuring a true zero-shot evaluation of the model's generalization capabilities.
Key Insights and Ideas
- The Bitter Lesson in Vision: The most significant finding is that architectural tweaks for specific tasks matter less than scale. As the authors increased parameter count (from 9B to 32B) and training data, the depth error dropped consistently without changing the underlying "generalist" recipe.
- Geometry as a Generative Prior: A model that knows how to generate a realistic scene must inherently understand its geometry. By forcing the model to generate RGB and Depth jointly, it develops a more robust spatial understanding than a model trained only to regress depth values.
- Preserving Generative Quality: Unlike previous attempts to repurpose generators, Iris-Z does not "forget" how to draw. It remains a high-quality text-to-image model while gaining state-of-the-art depth estimation skills.
Key Results
Iris-Z-big (32B) currently sets a new bar for generalist models in depth estimation:
- Benchmark Dominance: On Marigold’s benchmarks, Iris-Z-big achieved a mean AbsRel of 2.92, significantly better than the specialist MoGe-3 (3.12) and the previous generative leader Marigold V2 (4.10).
- Generalization: On MoGe-3’s own protocol, Iris-Z-big outperformed MoGe-3 on six out of seven datasets, with a mean error of 3.56 vs. 3.70.
- Efficiency of Scale: Even the smaller Iris-Z-small (9B) proved competitive, matching the performance of specialized state-of-the-art models.
Limitations and Future Work
Despite its success, Iris-Z has a few notable hurdles:
- Relative vs. Metric Depth: The current model predicts relative depth (scale and shift invariant). It doesn't yet provide absolute metric distances (e.g., "this wall is exactly 3.2 meters away"). The authors plan to address this by adding camera intrinsic tokens to the sequence.
- Computational Cost: Being a large generative model, Iris-Z is slow. One prediction takes about 15 seconds on an H200, compared to just 0.4 seconds for the specialist MoGe-3. The authors suggest that distillation techniques could eventually bridge this gap.
- Resolution: The model is currently trained for resolutions up to ~1024x1024, leaving room for improvement in ultra-high-resolution 4K/8K scenarios.
The team at SperidLabs views depth as just the first step. They intend to add more modalities—like surface normals and segmentation—to the same unified sequence, moving closer to a single, all-encompassing model for visual intelligence.
Summary updated