Learning Image Restoration from Internal Statistics

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2224 papers363 conceptsfrozen 2026-10-04

Deep learning typically relies on the "Big Data" paradigm: to learn how to fix a blurry or noisy image, a model must first see millions of examples of clean and corrupted pairs. However, a growing body of research suggests that a single image often contains enough information to train its own restoration model.

This direction, often called internal learning or zero-shot restoration, shifts the focus from external datasets to the unique statistical "DNA" of a single scene. By treating a single corrupted image as a complete dataset, researchers can restore images for which no ground truth exists—from rare medical scans to unique astronomical observations.

The Fractal Nature of Natural Images

The core intuition behind internal learning is that natural images are not random; they are highly redundant. Patterns, textures, and structures repeat themselves within the same image, often across different scales. A brick in the foreground of a photo looks remarkably like a smaller, slightly blurrier version of a brick in the background.

Assaf Shocher, Michal Irani, and their colleagues demonstrated this in InGAN, showing that a Generative Adversarial Network (GAN) can learn the "internal DNA" of a single image. By training on the distribution of patches within that one image, the model can synthesize new versions of the scene that preserve its unique local textures and structural logic. This internal redundancy is so powerful that it can even be used to represent data as a "Neural Collage"—a differentiable fractal representation that captures multi-scale patterns for high-resolution compression, as explored by Michael Poli and Winnie Xu in Self-similarity Priors.

Cross-Scale Self-Supervision

The most common way to exploit this internal redundancy is through cross-scale supervision. If we want to super-resolve a low-resolution (LR) image but lack a high-resolution (HR) target, we can create a "synthetic" training pair by downsampling the LR image even further.

  1. Downsample: Take the input image $I$ and create a smaller version $I_{down}$.
  2. Train: Train a network to map $I_{down}$ back to $I$.
  3. Inference: Apply that same learned mapping to the original image $I$ to produce a super-resolved version $I_{super}$.

This works because the relationship between the "coarse" and "fine" scales of an image is often consistent. Jérémy Scanvic leverages this in Scale-Equivariant Imaging, using scale-invariance as a universal prior to hallucinate missing high-frequency details. Similarly, Qinwen Huang and Ye Zhou applied this to cryo-electron microscopy in Cryo-ZSSR, allowing for near-atomic reconstructions without any external training data.

Blind-Spot Inversion and Noise Tolerance

Internal learning isn't limited to resolution; it is equally effective for denoising and deconvolution. The challenge here is distinguishing between the "signal" (the actual scene) and the "noise" (random corruption).

In Noise2Void, Alexander Krull and Florian Jug introduced a "blind-spot" architecture. By forcing a network to predict a pixel's value based only on its neighbors, the model learns to capture the spatially correlated signal while ignoring the zero-mean, uncorrelated noise.

Hirofumi Kobayashi and Löıc A. Royer extended this logic to Image Deconvolution, combining blind-spot self-supervision with the known physics of camera blur. Because the model is trained exclusively on the corrupted image it is meant to restore, it specializes in the specific noise distribution and structural priors of that individual scene, often outperforming classical algorithms like Lucy-Richardson.

Test-Time Adaptation: Bridging Big Data and Local Context

While purely internal models are powerful, they can be slow because they require training a new model for every image. A hybrid approach, test-time adaptation, uses a pre-trained model as a starting point and then "fine-tunes" it on the specific test image.

Zhangyang Wang and colleagues pioneered this in Self-tuned deep super resolution, where a model pre-trained on "big data" is dynamically adapted to the specific textures of a test input. This allows the network to leverage general visual knowledge while specializing in the unique, repeating patterns (like architectural grids or fabric textures) found in the target image.

Radu Timofte and others further validated this in Single Image Super Resolution - When Model Adaptation Matters, showing that even a few gradient steps on internal examples can significantly boost performance and eliminate the "domain gap" that occurs when a model encounters a type of image it hasn't seen during training.

Why Internal Learning Works

The success of these methods across super-resolution, denoising, and even shadow removal—as seen in the work of Dimitris Samaras and Hieu Le in From Shadow Segmentation to Shadow Removal—stems from three key advantages:

  1. No Domain Gap: The model is trained on the exact distribution it needs to process. There is no risk of the model being confused by synthetic noise that doesn't match real-world sensors.
  2. Zero Ground Truth: It enables restoration in fields like medical imaging or microscopy where "clean" versions of an image are physically impossible to capture.
  3. Adaptive Priors: Instead of a generic "average" of all natural images, the model learns a specific prior for the current scene, preserving sharp edges and unique textures that general models might smooth away.

By treating the image not just as a signal to be processed, but as a dataset to be learned, internal learning provides a path toward truly autonomous and adaptive computer vision.

Papers in this version

Showing 60 of 224 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 146 papers
  2. Version 1 · 2026-10-04 · 224 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_scale_self_similarity_224a9bc6,
  title = {cross-scale self-similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-scale-self-similarity/rdr_224a9bc6488f40128d4a5ec0768f38db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_image_restoration_from_internal_224a9bc6_v1,
  title = {Learning Image Restoration from Internal Statistics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-image-restoration-from-internal-statistics/rdr_224a9bc6488f40128d4a5ec0768f38db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}