In traditional digital imaging, processing is a relay race. A camera sensor captures raw data, an Image Signal Processing (ISP) pipeline demosaics and enhances it, and finally, a codec like JPEG or HEVC compresses it for storage. Each runner in this race operates in isolation, unaware of the others' needs.
Recent research suggests this "cascaded" approach is fundamentally inefficient. By "breaking the isolation" between compression and restoration, researchers are developing unified models that perform these tasks simultaneously. This shift from sequential to joint optimization allows systems to achieve higher visual quality at significantly lower bitrates.
The Sequential Bottleneck
The primary flaw in sequential pipelines is the accumulation of errors and the mismanagement of information. When an enhancement step (like denoising or low-light boosting) precedes compression, it often amplifies sensor noise. Standard codecs then treat this amplified noise as "detail," wasting precious bits to reconstruct artifacts that the viewer doesn't want to see. Conversely, if compression happens first, it may discard subtle textures that a subsequent restoration model would have needed to reconstruct a clean image.
Jian Jin and Xu Zou demonstrate this in the context of low-light photography. In Make Lossy Compression Meaningful for Low-Light Images, they show that treating enhancement and compression as independent steps leads to "blocky" artifacts and lost textures. By using a unified architecture, the model learns to enhance the image in a way that is inherently "compression-friendly," avoiding the amplification of noise before the data bottleneck.
Prioritizing Content over Degradation
A core intuition behind joint optimization is that a codec should only spend its "bit budget" on the underlying scene, not the degradations affecting it. Jiacheng Li and Zhiwei Xiong explored this in All-in-One Image Compression and Restoration. By integrating restoration directly into a variational autoencoder, the model learns to filter out rain, snow, or haze during the encoding process.
Because the encoder "knows" the restoration goal, it ignores weather artifacts and focuses on preserving the clean signal. This results in a dual benefit: the file size is smaller because noise isn't being encoded, and the final image is cleaner because the restoration is informed by the original raw features. Similarly, Yandong Guo and Charles A. Bouman showed that using a shared dictionary for both restoration and compression in binary document processing allows the system to use clean character patterns as both a denoising prior and a compression basis.
Compressing the Raw Signal
One of the most effective ways to improve efficiency is to defer "data expansion" until after the compression bottleneck. In a standard camera, demosaicing interpolates a single-channel Bayer pattern into a three-channel RGB image, effectively tripling the amount of data before it even reaches the compressor.
Researchers like Pascal Bacchus and Aline Roumy argue for Joint Compression and Demosaicking. By feeding raw sensor data directly into an encoder and performing color interpolation at the decoder, they avoid transmitting the redundant information created by interpolation. This approach is mirrored in the RAWtoBit network by Wooseok Jeong and Seung-Won Jung, which integrates the entire ISP and compression pipeline into a single heterogeneous architecture.
This principle also extends to specialized hardware. Bo Chen and colleagues developed RevSCI-net to reconstruct high-resolution video from compressed snapshots. By integrating demosaicing directly into the reconstruction pipeline, they recover RGB video from raw measurements in one step, preserving more detail than traditional two-stage methods.
Task-Aware Latent Spaces
The ultimate goal of joint optimization is often not just a "pretty" image for humans, but a "useful" image for machines. Xiufeng Xie and Ji Liu demonstrated this with AutoJPEG, which jointly optimizes JPEG quantization tables and neural network weights. This ensures the compression process preserves the specific frequency bands most critical for downstream AI tasks, like object detection.
For scenarios requiring both human and machine utility, Saeed Ranjbar Alvar and Ivan V. Bajic proposed a scalable latent space. Their framework splits data into a "base layer" for the clean image and an "enhancement layer" for the original noise. This allows a single bitstream to serve multiple purposes—providing a denoised version for viewing while retaining the raw data for scientific analysis. Similarly, Qi Zhang and Shiqi Wang introduced LL-ICM, which uses Large Vision-Language Models to ensure that compressed features remain optimized for low-level vision tasks like deraining.