Scaling Properties of Diffusion Models for Perceptual Tasks

Rahul Ravishankar, Zeeshan Patel, Jathushan Rajasegaran, Jitendra Malik

TL;DR

This paper demonstrates that diffusion models follow predictable scaling power laws for visual perception tasks like depth estimation and optical flow. By optimizing both training-time and test-time compute—specifically through ensembling and increased denoising steps—the authors achieve state-of-the-art performance using significantly less data and compute than previous generative-based perception methods.

What Problem Does This Paper Address?

While diffusion models have revolutionized image generation, their potential for "inverse vision" or perception tasks (e.g., "given this image, what is the depth?") is still being explored. Traditional perception models are often task-specific and require massive datasets.

The authors address two main questions:

  1. Do diffusion models for perception follow the same "scaling laws" (where performance improves predictably with more compute and parameters) seen in Large Language Models (LLMs)?
  2. Can we improve performance not just by training bigger models, but by letting the model "think longer" during inference?

Methods

The researchers treat perception tasks as a form of image-to-image translation. They use a Diffusion Transformer (DiT) backbone and fine-tune it to generate task-specific maps (like depth or flow) conditioned on an input RGB image.

Key technical components include:

  • Unified Framework: A single model architecture capable of handling multiple tasks by concatenating the RGB input with a noised version of the ground truth.
  • Scaling Training Compute: They experimented with model size (from 14.8M to 1.9B parameters), pre-training duration, and input resolution.
  • Mixture of Experts (MoE) & Upcycling: To increase model capacity without a linear increase in compute, they used MoE layers and "upcycling"—converting a pre-trained dense model into an MoE model to further boost performance.
  • Scaling Test-Time Compute: They exploited the iterative nature of diffusion by increasing the number of denoising steps and using test-time ensembling (generating multiple predictions from different noise seeds and merging them via median compilation).
Figure 1: A Unified Framework: We fine-tune a pre-trained Diffusion Model (DM), for visual perception tasks. We take a RGB image, and a conditional image (i.e. next video frame, occlusion mask, etc.), along with the noised image of the ground truth prediction. Our model generates pre- dictions for visual tasks such as depth estimation, optical flow prediction, and amodal segmentation, based on the conditional task embedding. We train a generalist model that can perform all three tasks with exceptional performance.

Datasets

The models were evaluated across several standard benchmarks:

  • Pre-training: ImageNet-1K.
  • Depth Estimation: Trained on Hypersim; evaluated on ETH3D, NYUv2, ScanNet, and DIODE.
  • Optical Flow: Trained and evaluated on FlyingChairs.
  • Amodal Segmentation: Trained on Pix2Gestalt; evaluated on COCO-A and MP3D.

Key Insights and Ideas

  • Perception Power Laws: The paper empirically establishes that perception tasks follow clear power-law scaling. As you increase compute (MACs), the error rates drop following a predictable curve ($L(C) = a \cdot C^b$).
  • "Thinking" at Inference Time: Similar to OpenAI’s o1 model in the LLM space, this work shows a trade-off between training and inference. Allocating more compute to the denoising process at test-time can compensate for a smaller training budget.
  • Resolution is a Potent Scaler: Increasing image resolution (and thus the number of tokens) is one of the most effective ways to scale compute and improve downstream accuracy.
  • Upcycling is "Free" Performance: Converting a fine-tuned dense model to an MoE model (upcycling) provided an average improvement of 5.3% in Absolute Relative Error for depth tasks.
Figure 2: Scaling at Model Size: For genera- tive pre-training of DiT models, we observe clear power law scaling behavior as we increase the model size.

Key Results

  • Depth Estimation: Their model matches the performance of Marigold (a leading diffusion-based depth estimator) on the Hypersim dataset and actually outperforms it on ETH3D. Crucially, they achieved this while using three orders of magnitude less pre-training data.
  • Optical Flow: The diffusion-based approach achieved an End-Point Error (EPE) of 3.08 on FlyingChairs, proving competitive with specialized architectures like DeepFlow and FlowNet.
  • Generalist Capability: They successfully trained a single "unified expert" model that performs depth, flow, and amodal segmentation simultaneously with high accuracy.
Scaling laws observed during fine-tuning for depth estimation.

Limitations and Future Work

  • Inference Latency: While scaling test-time compute improves accuracy, it makes the model significantly slower, as it requires multiple denoising passes and ensembling. This makes real-time application challenging.
  • Synthetic Data Reliance: The training relies heavily on high-quality synthetic datasets (like Hypersim). The authors suggest that future work could explore scaling with more diverse, real-world data.
  • Task Expansion: The current work focuses on 2D and 2.5D tasks; extending these scaling recipes to full 3D scene understanding or video-consistent perception is a promising next step.

Summary updated

Abstract

In this paper, we argue that iterative computation with diffusion models offers a powerful paradigm for not only generation but also visual perception tasks. We unify tasks such as depth estimation, optical flow, and amodal segmentation under the framework of image-to-image translation, and show how diffusion models benefit from scaling training and test-time compute for these perceptual tasks. Through a careful analysis of these scaling properties, we formulate compute-optimal training and inference recipes to scale diffusion models for visual perception tasks. Our models achieve competitive performance to state-of-the-art methods using significantly less data and compute. To access our code and models, see https://scaling-diffusion-perception.github.io .
BibTeX
@inproceedings{ravishankar2025scaling,
  author = {Rahul Ravishankar and Zeeshan Patel and Jathushan Rajasegaran and Jitendra Malik},
  title = {{Scaling Properties of Diffusion Models for Perceptual Tasks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01208},
}
Figure