For decades, computer vision and image restoration models have been fed standard, 8-bit sRGB images—what Ljungbergh et al. describe as "cooked" data. These images look visually pleasing to humans because a camera's internal Image Signal Processor (ISP) has carefully tuned their contrast, colors, and sharpness. However, this aesthetic tuning comes at a steep cost: it permanently destroys the underlying physical properties of light.
An emerging paradigm shift argues that for tasks like denoising, super-resolution, and low-light enhancement, we should bypass the ISP entirely. By operating directly on the linear, high-bit-depth RAW data captured by the camera sensor, models can preserve critical signal information that is otherwise lost forever.
The Information Bottleneck of the Traditional ISP
To understand why RAW-domain processing works, we must first look at how a traditional ISP alters sensor data. A camera sensor is essentially a photon counter covered by a color filter array (most commonly a Bayer mosaic), where each pixel records only a single color channel (Red, Green, or Blue) in a strictly linear relationship with scene radiance.
To turn this sparse grid into a standard image, the ISP executes a sequence of complex operations:
- Demosaicing: Interpolating the missing color values at each pixel location.
- Denoising and Sharpening: Applying non-linear spatial filters to smooth out grain and sharpen edges.
- Tone Mapping and Gamma Correction: Compressing the high dynamic range of the real world into a non-linear curve suitable for 8-bit displays.
- Quantization and Compression: Saving the final output as a compressed format like JPEG.
[Scene Radiance]
│
▼
[Linear RAW Data] ◄─── RAW-Domain Processing (Preserves physical signal)
│
┌─────┴────────────────────────────────┐
│ Traditional ISP Pipeline │
│ • Demosaicing (Spreads noise) │
│ • Non-linear Tone Mapping │
│ • 8-bit Quantization (Clips data) │
└─────┬────────────────────────────────┘
│
▼
[Cooked sRGB Image] ◄─── Traditional Processing (Distorted, lossy signal)
While these steps optimize images for human eyes, they act as an information bottleneck for neural networks. As Nguyen et al. and Yang et al. note, non-linear tone curves permanently clip highlights and crush shadow details into unrecoverable black pixels.
Furthermore, Jang et al. point out that operations like demosaicing and sharpening alter the statistical distribution of sensor noise. What began as independent, predictable photon noise becomes spatially correlated, non-linear distortion. Trying to denoise or restore an image after these steps is an uphill battle because the network must invert a series of lossy, black-box transformations.
The Mathematical Power of Linearity
The primary reason RAW-domain processing succeeds is linearity. In the RAW domain, pixel values are directly proportional to the number of photons hitting the sensor (Ouyang et al.). This linear relationship aligns perfectly with the physics of light accumulation, making complex image degradation problems mathematically straightforward to invert.
Consider the task of reflection removal. When shooting through a glass window, the captured image ($M$) is a physical mixture of the transmitted scene ($T$) and the reflection ($R$). In the RAW domain, this is a clean linear combination:
$$M = T + R$$
As Lei et al. demonstrate, neural networks can separate these layers far more accurately when the underlying data preserves this strict physical linearity, whereas post-ISP non-linearities warp the relationship and blend the layers inextricably.
A similar principle applies to motion blur and lens degradations. As explored by Fang et al. and Xu et al., blur operates as a linear convolution on the sensor. Inverting this convolution in a non-linear intensity space introduces severe mathematical inaccuracies, leading to visual artifacts like ringing. Performing deblurring or super-resolution in the linear irradiance domain ensures that the optimization process mirrors the physical accumulation of light.
Unlocking High Dynamic Range and Shadow Details
Standard sRGB images compress data into 8 bits per channel, providing only 256 discrete levels of intensity. In contrast, RAW sensor data typically retains 12 to 14 bits of precision (4,096 to 16,384 levels).
In extreme low-light photography, this extra bit depth is the difference between visibility and total information loss. In their seminal work on extreme low-light imaging, Chen et al. showed that by packing raw Bayer patterns directly into multi-channel inputs, deep networks can reconstruct clear, color-accurate images from short-exposure frames that appear completely pitch-black in sRGB format.
Bayer Pattern (RAW) Multi-Channel Tensor
┌───┬───┬───┬───┐ ┌───┬───┐ ┌───┬───┐
│ R │ G │ R │ G │ │ R │ R │ │ G │ G │
├───┼───┼───┼───┤ ├───┼───┤ ├───┼───┤
│ G │ B │ G │ B │ ───────────► │ R │ R │ │ G │ G │
├───┼───┼───┼───┤ └───┴───┘ └───┴───┘
│ R │ G │ R │ G │ Red Channel Green-1 Channel
└───┴───┴───┴───┘ (Half spatial resolution, original bit-depth)
By operating on the raw sensor grid before demosaicing, models exploit the precise spatial and spectral correlations of the sensor without introducing interpolation artifacts (Mildenhall et al., Zhou et al.).
Because short-exposure RAW data inherently avoids highlight clipping, models designed by Yan et al. and Meng et al. can pull vivid details out of deep shadows while maintaining pristine highlights. This allows for single-shot ultra-high dynamic range (UHDR) reconstruction, bypassing the ghosting artifacts that plague traditional multi-exposure burst methods.
Simplifying Downstream Computer Vision
The benefits of linear signal preservation extend far beyond computational photography; they fundamentally improve how machines "see."
Standard ISPs are tuned to make images look aesthetically pleasing to humans, often applying heavy denoising that smears fine textures or aggressive contrast adjustments that destroy global context. For computer vision models, this "cooking" process removes vital cues.
For instance, traditional feature detectors and matchers frequently fail in low-light environments because the ISP's denoising stages erase high-frequency structural landmarks. By training feature descriptors directly on RAW sensor data, Zhao et al. and Suganuma et al. demonstrated that algorithms can find reliable, sub-pixel keypoints even in near-total darkness.
Similarly, for high-level tasks like shadow detection, Sun et al. showed that preserving the linear relationship of scene radiance prevents networks from confusing dark-colored surfaces with actual shadows—a frequent failure mode in sRGB space where global contrast is sacrificed for local visibility.
By treating the camera sensor not as a generator of pretty pictures, but as a precise physical instrument for measuring light, RAW-domain processing provides a cleaner, richer, and mathematically sound foundation for the next generation of computer vision and image restoration models (Wen et al.).