Restoring Images Without Paired Real World Data

This direction is merged into another direction: paired data scarcity (real-world paired data scarcity). Its last version stays readable and citable here.

Version 1 of 151 papers51 conceptsfrozen 2026-10-04

In the world of deep learning, "ground truth" is the ultimate teacher. To train a model to remove rain, haze, or underwater murkiness, we typically need a dataset of pairs: a degraded image (the input) and a perfectly clear version of the exact same scene (the target).

But here lies a physical impossibility. You cannot capture a photo of a city in a thick fog and, at the same exact moment, capture it in crystal-clear air. By the time the fog clears, the sun has moved, the shadows have shifted, and the cars have driven away. This lack of perfectly aligned, real-world paired data is the fundamental bottleneck in image restoration.

The Synthetic Trap and the Domain Gap

To circumvent this, researchers traditionally rely on synthetic data. We take a clear image and use mathematical models—like the Koschmieder light scattering model—to "add" artificial haze or rain.

However, as Yuanjie Shao and colleagues point out in Domain Adaptation for Image Dehazing, these models are often too simplistic. Real-world degradation is a "complex cocktail" of varying particle sizes and lighting conditions. When a model trained only on "fake" rain meets the messy physics of a real storm, it often fails to generalize. This discrepancy is known as the domain gap.

Strategy 1: Semi-Supervised Bridging

If we have plenty of "fake" pairs but only "unpaired" real images, how do we learn? One powerful approach is semi-supervised learning.

In Semi-supervised CNN for Single Image Rain Removal, Deyu Meng and his team propose a framework that trains on both. The network learns the basic "how-to" of deraining from synthetic labels, while simultaneously adapting to the statistical distribution of real-world rain. Similarly, SJDL-Vehicle by Wei-Ting Chen uses this dual-stream approach to ensure that a model can both defog an image and identify a vehicle within it, even when no "clean" version of that specific foggy car exists.

Strategy 2: Generative Domain Translation

Another solution is to treat restoration as a "translation" problem between two domains: the "degraded" domain and the "clean" domain.

Generative Adversarial Networks (GANs) are particularly adept at this. Kenji Enomoto and others introduced McGANs, which use multispectral data (like Near-Infrared) to help the generator "see through" thin clouds. Because the discriminator only cares if the final image looks like a real clear image, the model can learn to remove artifacts without needing a pixel-perfect ground truth for every training sample.

More recently, researchers have moved toward Contrastive Learning. In Unpaired Deep Image Deraining Using Dual Contrastive Learning, the model is taught to "pull" a restored image toward the distribution of clean photos while "pushing" it away from the distribution of rainy ones. This allows the network to learn the concept of "cleanness" without needing a specific reference photo.

Strategy 3: Physics as a Prior

If we don't have a teacher (ground truth), we can use the laws of physics as a guide. This is the core of Zero-Shot and Unsupervised restoration.

In Zero-Shot Single Image Restoration, Aupendu Kar and his team show that by perturbing the mathematical parameters of light scattering models, a network can learn to restore a single image from scratch. It doesn't need a dataset; it just needs to understand how light and haze interact.

Similarly, the CSUD framework leverages a "Channel Consistency Prior"—the observation that rain streaks usually look similar across Red, Green, and Blue channels. By enforcing this consistency during training, the model can identify and remove rain streaks in an unsupervised fashion, often outperforming supervised models that were "overfit" to synthetic data.

The Future: Self-Supervision and Diffusion

The field is currently shifting toward even more robust self-supervision. NightHaze by Beibei Lin and Robby T. Tan uses "severe augmentation"—intentionally making clear images look even worse—to force the model to learn deep structural priors.

Meanwhile, methods like Osmosis are using Diffusion Priors. By training a diffusion model on clear, in-air images, researchers can "guide" the restoration of underwater images. The model knows what a natural scene should look like, and it uses that internal knowledge to "denoise" the murkiness of the ocean.

By moving away from the requirement of paired data, these researchers are finally allowing computer vision to step out of the laboratory and into the unpredictable, messy reality of the natural world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_restoring_images_without_paired_real_wor_aac6e6cf,
  title = {Restoring Images Without Paired Real World Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/restoring-images-without-paired-real-world-data/rdr_aac6e6cfa8634f8bbaf6f16debab0a9c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_restoring_images_without_paired_real_wor_aac6e6cf_v1,
  title = {Restoring Images Without Paired Real World Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/restoring-images-without-paired-real-world-data/rdr_aac6e6cfa8634f8bbaf6f16debab0a9c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}