TL;DR
This paper demonstrates that diffusion models follow predictable scaling power laws for visual perception tasks like depth estimation and optical flow. By optimizing both training-time and test-time compute—specifically through ensembling and increased denoising steps—the authors achieve state-of-the-art performance using significantly less data and compute than previous generative-based perception methods.
What Problem Does This Paper Address?
While diffusion models have revolutionized image generation, their potential for "inverse vision" or perception tasks (e.g., "given this image, what is the depth?") is still being explored. Traditional perception models are often task-specific and require massive datasets.
The authors address two main questions:
- Do diffusion models for perception follow the same "scaling laws" (where performance improves predictably with more compute and parameters) seen in Large Language Models (LLMs)?
- Can we improve performance not just by training bigger models, but by letting the model "think longer" during inference?
Methods
The researchers treat perception tasks as a form of image-to-image translation. They use a Diffusion Transformer (DiT) backbone and fine-tune it to generate task-specific maps (like depth or flow) conditioned on an input RGB image.
Key technical components include:
- Unified Framework: A single model architecture capable of handling multiple tasks by concatenating the RGB input with a noised version of the ground truth.
- Scaling Training Compute: They experimented with model size (from 14.8M to 1.9B parameters), pre-training duration, and input resolution.
- Mixture of Experts (MoE) & Upcycling: To increase model capacity without a linear increase in compute, they used MoE layers and "upcycling"—converting a pre-trained dense model into an MoE model to further boost performance.
- Scaling Test-Time Compute: They exploited the iterative nature of diffusion by increasing the number of denoising steps and using test-time ensembling (generating multiple predictions from different noise seeds and merging them via median compilation).
Datasets
The models were evaluated across several standard benchmarks:
- Pre-training: ImageNet-1K.
- Depth Estimation: Trained on Hypersim; evaluated on ETH3D, NYUv2, ScanNet, and DIODE.
- Optical Flow: Trained and evaluated on FlyingChairs.
- Amodal Segmentation: Trained on Pix2Gestalt; evaluated on COCO-A and MP3D.
Key Insights and Ideas
- Perception Power Laws: The paper empirically establishes that perception tasks follow clear power-law scaling. As you increase compute (MACs), the error rates drop following a predictable curve ($L(C) = a \cdot C^b$).
- "Thinking" at Inference Time: Similar to OpenAI’s o1 model in the LLM space, this work shows a trade-off between training and inference. Allocating more compute to the denoising process at test-time can compensate for a smaller training budget.
- Resolution is a Potent Scaler: Increasing image resolution (and thus the number of tokens) is one of the most effective ways to scale compute and improve downstream accuracy.
- Upcycling is "Free" Performance: Converting a fine-tuned dense model to an MoE model (upcycling) provided an average improvement of 5.3% in Absolute Relative Error for depth tasks.
Key Results
- Depth Estimation: Their model matches the performance of Marigold (a leading diffusion-based depth estimator) on the Hypersim dataset and actually outperforms it on ETH3D. Crucially, they achieved this while using three orders of magnitude less pre-training data.
- Optical Flow: The diffusion-based approach achieved an End-Point Error (EPE) of 3.08 on FlyingChairs, proving competitive with specialized architectures like DeepFlow and FlowNet.
- Generalist Capability: They successfully trained a single "unified expert" model that performs depth, flow, and amodal segmentation simultaneously with high accuracy.
Limitations and Future Work
- Inference Latency: While scaling test-time compute improves accuracy, it makes the model significantly slower, as it requires multiple denoising passes and ensembling. This makes real-time application challenging.
- Synthetic Data Reliance: The training relies heavily on high-quality synthetic datasets (like Hypersim). The authors suggest that future work could explore scaling with more diverse, real-world data.
- Task Expansion: The current work focuses on 2D and 2.5D tasks; extending these scaling recipes to full 3D scene understanding or video-consistent perception is a promising next step.
Summary updated