Most image restoration happens at the very end of the imaging pipeline. We take a finished photo or a reconstructed medical scan and try to "fix" the noise. However, a growing body of research suggests that this is fundamentally the wrong time to act. By the time an image reaches the sRGB or reconstruction stage, the original sensor noise has been warped, smeared, and amplified by a series of non-linear transformations.
The emerging consensus across photography and medical imaging is to move restoration to the Raw domain—the linear, unprocessed data captured directly by the sensor.
The ISP Trap: Why Processed Images are Hard to Clean
In a standard camera, the Image Signal Processor (ISP) performs several steps to turn raw sensor readings into a pretty picture: demosaicing (interpolating colors), white balance, gamma correction, and tone mapping. While these make images look good to humans, they are a nightmare for denoising algorithms.
As Tim Brooks, Ben Mildenhall, and their colleagues argue in Unprocessing Images for Learned Raw Denoising, these non-linear operations distort noise statistics. In the RAW domain, noise is mathematically predictable—usually a combination of Gaussian and Poisson distributions. Once it passes through the ISP, that noise becomes spatially correlated and non-Gaussian.
Abdelrahman Abdelhamed and Michael S. Brown, through the NTIRE 2019 and 2020 challenges, demonstrated that denoising raw sensor data yields significantly higher fidelity. Because the signal is still linear, models can distinguish between actual light (signal) and sensor error (noise) much more accurately.
Preventing Artifact Spreading
The problem isn't just that noise becomes "complex"; it's that the pipeline physically spreads the error. Consider demosaicing, where a camera interpolates missing color values from neighboring pixels. If one pixel has a noise spike, demosaicing spreads that spike to its neighbors across all three color channels.
Huanjing Yue and Yijia Cheng highlight this in their work on Recaptured Raw Screen Image and Video Demoiréing. They found that moiré patterns—interference ripples from digital screens—are much easier to remove in RAW because they are isolated to specific color channels. Once demosaiced, these patterns become "baked in" across the entire image, making them nearly impossible to untangle.
This principle extends to 3D data as well. Gene Cheung and Xue Zhang show in Graph-Based Depth Denoising that cleaning depth data in the 2D image domain before projecting it into 3D space prevents measurement errors from being distributed across multiple spatial coordinates.
Stopping the Amplification in Medical Imaging
In medical contexts like CT or PET scans, the "raw" data is a sinogram—a collection of X-ray projections. The process of turning these projections into a 3D cross-section is called reconstruction.
The danger here is mathematical amplification. Any noise present in the raw projection data is amplified by the reconstruction algorithm, turning random speckles into structured artifacts like streaks or bands. Long Zhou and colleagues developed Noise2Projection to solve this by denoising the raw X-ray data first. By cleaning the signal at the detector level, they ensure the reconstruction algorithm has "clean" inputs, preventing the birth of those distracting streaks.
Similarly, the TriPLET framework by Caiwen Jiang and Dinggang Shen uses a triple-domain approach, but emphasizes that denoising the projection domain is the critical first step to prevent noise propagation.
The Order of Operations
While the "denoise first" rule is dominant, some researchers explore more nuanced sequences. Yu Guo, Jean-michel Morel, and Gabriele Facciolo have investigated joint demosaicing-denoising. They suggest that for moderate noise, demosaicing first can actually preserve high-frequency details that a raw denoiser might accidentally smooth away.
However, even in these cases, the focus remains on the early pipeline. Whether it's Hao Guan's NODE network decomposing noise into sensor-specific streams or Xu Ji's SeNAS-Net using self-supervised learning on projections, the intuition is the same:
Fix the data where the physics is simplest. By operating on the raw sensor measurements, we avoid the "correlation trap" of processed images and stop noise from evolving into something much harder to kill.