When you take a photo with a smartphone, the device does not simply record the light hitting its sensor. Instead, an Image Signal Processor (ISP) performs a series of complex, non-linear transformations—tone mapping, white balancing, and gamma correction—to produce a visually pleasing sRGB image. While these steps make photos look "right" to human eyes, they destroy the mathematical relationship between pixel values and physical scene radiance.
For computer vision algorithms that rely on the physics of light, this aesthetic processing is a source of significant error. A growing body of research focuses on "unprocessing" or "de-rendering" these images to recover the original linear radiance of the scene.
The Linearity Gap
The fundamental problem is that physical processes like light addition, blurring, and reflection occur in a linear space. If two light sources hit a surface, their intensities add up linearly. However, once a camera applies a non-linear Camera Response Function (CRF), that simple addition is warped.
As Zhixiang Chi points out in the context of reflection removal, synthesizing training data by mixing sRGB images leads to unrealistic artifacts. This is because the underlying physics of light addition happens before the camera's non-linear processing. Similarly, Li Ma demonstrates in Deblur-NeRF that modeling motion blur in a gamma-corrected space creates a mismatch with reality, as blur is physically a linear combination of light intensities over time.
The Camera Response Function
To bridge the gap between "pretty" pixels and physical light, researchers must estimate the Camera Response Function (CRF). This function represents the mapping from the irradiance at the sensor to the final digital values.
In High Dynamic Range Image Reconstruction, Asla Medeiros e Sá and Paulo Cezar Pinto Carvalho describe how reversing this function is the key to High Dynamic Range (HDR) imaging. If we can't linearize the image, we can't accurately merge multiple exposures or recover the true brightness of a scene.
Estimating this function is difficult because it is often proprietary and varies between manufacturers. Pedro Rodrigues and João P. Barreto have developed methods to estimate the CRF from a single image by exploiting specific lighting conditions, such as the vignetting found in medical endoscopy. In the field of 3D reconstruction, Wiennat Mongkulmann shows that "auto-radiometric calibration" can simultaneously recover the CRF and the 3D shape of an object by ensuring the pixel values remain consistent with the physics of light reflection.
Unprocessing and De-rendering
If the goal is to perform high-precision tasks like denoising or super-resolution, it is often better to work directly in the RAW domain. Huanjing Yue and colleagues argue that RAW data, with its high bit-depth and linear response, provides a much cleaner signal for deep learning models than processed sRGB.
However, we don't always have access to RAW files. This has led to the development of "de-rendering" frameworks. Seonghyeon Nam and Michael S. Brown introduced a method in Learning sRGB-to-Raw-RGB De-rendering with Content-Aware Metadata that uses a tiny amount of metadata (1.5% of pixels) to reconstruct the full linear RAW image from a standard sRGB file.
Similarly, Nguyen Ho Man Rang and Michael S. Brown have shown that even a 64 KB overhead is enough to make a JPEG "self-contained" for RAW reconstruction, allowing vision algorithms to bypass the "lies" of the ISP.
From Pixels to Physical Units
The ultimate goal of this direction is to move beyond relative brightness to absolute physical units. Christophe Bolduc and Jean-François Lalonde have pushed this further with the Laval Photometric Indoor HDR Dataset, which provides images calibrated to actual luminance ($cd/m^2$).
When we treat an image as a collection of physical measurements rather than just a grid of colors, tasks that were once "impossible"—like predicting the exact lighting of a room from a single photo or performing Inverse Rendering of Indoor Scenes (Chih-Hao Lin et al.)—become mathematically tractable. By inverting the camera pipeline, we restore the link between the digital image and the physical world it represents.