RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation

Christoph Reinders, Radu Berdan, Beril Besbinar, Junji Otsuka, Daisuke IsoWACV 2025

TL;DR

This paper introduces RAW-Diffusion, a novel diffusion model that generates high-fidelity RAW images from RGB images. This approach significantly reduces the need for expensive, sensor-specific RAW data collection, enabling the creation of custom RAW datasets and improving downstream tasks like object detection, even with minimal training data.

What Problem Does This Paper Address?

Modern computer vision heavily relies on RGB images, which are processed and compressed from the original RAW sensor data. This processing, while making images human-friendly, inevitably leads to information loss and artifacts. RAW images, on the other hand, retain richer, unprocessed sensor information crucial for tasks requiring precise scene irradiance, especially in challenging conditions like low light. However, acquiring large, diverse RAW datasets for specific camera sensors is prohibitively expensive and labor-intensive. Existing methods for converting RGB to RAW often require sensor-specific configurations or large datasets. This paper addresses the challenge of generating high-quality RAW images from readily available RGB data in a data-efficient and flexible manner.

Methods

The core of RAW-Diffusion is a diffusion model adapted for RAW image generation. It leverages an RGB-guidance module to extract features from input RGB images. These features are then integrated into the diffusion process using novel "RGB-guided residual blocks" within a U-Net architecture. This guidance helps the model denoise a noisy RAW image iteratively, ultimately reconstructing a high-fidelity RAW output.

Key methodological contributions include:

  • RGB-Guidance Module: This module extracts relevant features from RGB images, acting as a guide for the RAW generation process.
  • RGB-Guided Residual Blocks: These blocks inject the extracted RGB features into the diffusion model's architecture, modulating the feature transformations to steer the RAW image reconstruction. This is a departure from simpler concatenation methods.
  • Direct RAW Prediction: Unlike some diffusion models that predict noise, RAW-Diffusion directly predicts the denoised RAW image.
  • Data Efficiency: The model is designed to achieve state-of-the-art performance with very few training samples, making it practical for new camera sensors.

Datasets

The authors utilize several datasets for training and evaluation:

  • FiveK Dataset: Used for training and evaluating RAW reconstruction.
  • NOD (Night Object Detection) Dataset: Consists of RAW images with object annotations, used for both RAW reconstruction and downstream object detection tasks.
  • Cityscapes and BDD100K: Large-scale RGB datasets that are converted to RAW format using RAW-Diffusion to create new, large-scale synthetic RAW datasets for downstream tasks.

Key Insights and Ideas

  • RGB as a Powerful Prior for RAW Generation: The paper demonstrates that RGB images, despite their inherent information loss, contain sufficient structural and semantic information to effectively guide the reconstruction of high-fidelity RAW images.
  • Data Efficiency is Achievable: The success of RAW-Diffusion with as few as 25 training samples is a significant finding, drastically lowering the barrier to entry for creating sensor-specific RAW data.
  • Bridging the Gap to Downstream Tasks: By generating realistic RAW data, RAW-Diffusion can facilitate training downstream models (like object detectors) on sensor-specific RAW data, potentially improving performance in challenging conditions.
  • RGB-Guided Residual Blocks are Key: The novel residual blocks are crucial for effectively integrating RGB guidance, outperforming simpler concatenation methods.

Key Results

  • State-of-the-Art RAW Reconstruction: RAW-Diffusion achieves top performance in terms of PSNR and SSIM on multiple DSLR camera datasets, outperforming existing methods. For instance, on the NOD Nikon dataset, it achieves a PSNR of 44.93, significantly higher than other methods.
  • Exceptional Data Efficiency: The model achieves comparable performance to training on the full dataset even when trained with only 25 samples. In some cases, performance even slightly improves with smaller datasets due to training dynamics.
  • Improved Object Detection: When used to generate synthetic RAW datasets (Cityscapes-RAW, BDD100K-RAW), RAW-Diffusion significantly boosts object detection performance (using Faster R-CNN and YOLOv8) on the NOD dataset, especially when combined with a small amount of original NOD data. For example, using RAW+BDD100K-RAW (ours) with Faster R-CNN on NOD Nikon yields an AP of 26.5, compared to 19.1 for RGB only.
  • Qualitative Improvements: Visual results show that RAW-Diffusion generates more detailed and artifact-free RAW images compared to other methods.

Limitations and Future Work

  • Camera-Specific Training: The current method requires dedicated training for each camera sensor. The authors plan to explore generalization abilities through multi-sensor conditioning.
  • Bias Mitigation: As a generative model, RAW-Diffusion cannot inherently mitigate existing dataset biases. Future work may address this.
  • Downstream Model Limitations: For object detection, limitations in standard models for handling small or overlapping objects in challenging scenes are noted, suggesting the need for specialized models.
  • Zero-Shot Performance: While promising, zero-shot object detection using generated datasets (e.g., Cityscapes-RAW) showed limitations when the source domain (daylight images) was too different from the target domain (NOD).

The paper suggests future work on multi-sensor generalization and addressing dataset biases.

Summary updated

BibTeX
@inproceedings{reinders2025raw,
  author = {Christoph Reinders and Radu Berdan and Beril Besbinar and Junji Otsuka and Daisuke Iso},
  title = {{RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure