Abstract
Deep Internal Learning (DIL) leverages the recurrence of patches within a single natural image to learn its internal statistical distribution. While previous methods predominantly rely on Generative Adversarial Networks (GANs), they often face challenges such as training instability and mode collapse. We present PatchGenCN, a novel framework that represents an image's internal distribution through a pyramid of Energy-Based Models (EBMs). By employing a bottom-up Convolutional Neural Network (ConvNet) to define an energy function and utilizing Langevin dynamics for synthesis, PatchGenCN explicitly models patch statistics across multiple scales. Our approach simplifies the DIL architecture by removing the need for a discriminator and achieves state-of-the-art results in image synthesis, super-resolution, and harmonization.
Introduction
Most deep learning paradigms require vast datasets to generalize effectively. However, natural images possess a unique property: small patches recur frequently at different locations and scales. Deep Internal Learning (DIL) exploits this redundancy to train models on a single image. While GAN-based models like SinGAN have shown promise, they are often difficult to tune and can struggle to maintain global structural coherence.
PatchGenCN addresses these issues by shifting from implicit generative modeling to explicit energy-based modeling. Instead of training a generator to fool a discriminator, we train a single ConvNet at each scale to assign low energy to patches from the training image and high energy to others. This unified framework allows for a more direct representation of the image's "DNA."
Methodology: The PatchGenCN Framework
PatchGenCN utilizes a coarse-to-fine pyramid of EBMs. At each level, the model learns the patch distribution of the image at a specific resolution.
Energy-Based Modeling and Analysis-by-Synthesis
The core of the model is a bottom-up ConvNet that maps an image to a scalar energy value. Training follows an "analysis-by-synthesis" scheme:
- Synthesis: Starting from noise (or an upsampled result from a coarser scale), the model uses Langevin dynamics (MCMC sampling) to iteratively refine the image toward lower energy states.
- Analysis: The network weights are updated to minimize the energy of the original training image while maximizing the energy of the synthesized samples.
Coarse-to-Fine Pyramid
To capture both global layout and fine textures, training proceeds across multiple scales. The synthesized output from a lower resolution serves as the starting point for the next level, significantly improving sampling efficiency and structural consistency.
Self-Supervised Parallel Training
To stabilize the EBM training, the authors introduce a dual-objective approach. The model is trained simultaneously on random synthesis and a self-supervised task, such as super-resolution. This ensures the learned energy function captures descriptive features useful for various downstream applications.
Experimental Results
The researchers evaluated PatchGenCN across diverse tasks, including random synthesis, image harmonization, and super-resolution.
Realism and Statistical Fidelity
In human perceptual studies, PatchGenCN's synthesized images were mistaken for real ones in 33-40% of trials, matching or exceeding the performance of GAN-based competitors. Quantitatively, the model achieved a Single Image Frechet Inception Distance (SIFID) of 0.09, outperforming SinGAN (0.11). This indicates that PatchGenCN more accurately captures the patch-level statistics of the source image.
Super-Resolution Performance
PatchGenCN demonstrates significant advantages in internal super-resolution. By learning the specific textures of a single image, it produces results with fewer artifacts and more realistic fine details compared to general-purpose or other internal learning models like ZSSR and DIP.
Conclusion
PatchGenCN provides a robust and theoretically grounded alternative to GANs for single-image learning. By explicitly modeling internal statistics through an energy-based framework, it achieves superior structural coherence and competitive realism. While the iterative nature of MCMC sampling results in slower inference compared to feed-forward generators, the model's simplicity, reduced parameter count (~0.99M), and versatility across tasks make it a significant advancement in deep internal learning.
Summary updated