Patchwise Generative ConvNet: Training Energy-Based Models From a Single Natural Image for Internal Learning

Zilong Zheng, Jianwen Xie, Ping Li

Abstract

Deep Internal Learning (DIL) leverages the recurrence of patches within a single natural image to learn its internal statistical distribution. While previous methods predominantly rely on Generative Adversarial Networks (GANs), they often face challenges such as training instability and mode collapse. We present PatchGenCN, a novel framework that represents an image's internal distribution through a pyramid of Energy-Based Models (EBMs). By employing a bottom-up Convolutional Neural Network (ConvNet) to define an energy function and utilizing Langevin dynamics for synthesis, PatchGenCN explicitly models patch statistics across multiple scales. Our approach simplifies the DIL architecture by removing the need for a discriminator and achieves state-of-the-art results in image synthesis, super-resolution, and harmonization.

Introduction

Most deep learning paradigms require vast datasets to generalize effectively. However, natural images possess a unique property: small patches recur frequently at different locations and scales. Deep Internal Learning (DIL) exploits this redundancy to train models on a single image. While GAN-based models like SinGAN have shown promise, they are often difficult to tune and can struggle to maintain global structural coherence.

PatchGenCN addresses these issues by shifting from implicit generative modeling to explicit energy-based modeling. Instead of training a generator to fool a discriminator, we train a single ConvNet at each scale to assign low energy to patches from the training image and high energy to others. This unified framework allows for a more direct representation of the image's "DNA."

Methodology: The PatchGenCN Framework

PatchGenCN utilizes a coarse-to-fine pyramid of EBMs. At each level, the model learns the patch distribution of the image at a specific resolution.

Energy-Based Modeling and Analysis-by-Synthesis

The core of the model is a bottom-up ConvNet that maps an image to a scalar energy value. Training follows an "analysis-by-synthesis" scheme:

  1. Synthesis: Starting from noise (or an upsampled result from a coarser scale), the model uses Langevin dynamics (MCMC sampling) to iteratively refine the image toward lower energy states.
  2. Analysis: The network weights are updated to minimize the energy of the original training image while maximizing the energy of the synthesized samples.

Coarse-to-Fine Pyramid

To capture both global layout and fine textures, training proceeds across multiple scales. The synthesized output from a lower resolution serves as the starting point for the next level, significantly improving sampling efficiency and structural consistency.

Comparison of multi-scale generation between SinGAN and PatchGenCN, highlighting superior structural preservation in the latter.

Self-Supervised Parallel Training

To stabilize the EBM training, the authors introduce a dual-objective approach. The model is trained simultaneously on random synthesis and a self-supervised task, such as super-resolution. This ensures the learned energy function captures descriptive features useful for various downstream applications.

Experimental Results

The researchers evaluated PatchGenCN across diverse tasks, including random synthesis, image harmonization, and super-resolution.

Realism and Statistical Fidelity

In human perceptual studies, PatchGenCN's synthesized images were mistaken for real ones in 33-40% of trials, matching or exceeding the performance of GAN-based competitors. Quantitatively, the model achieved a Single Image Frechet Inception Distance (SIFID) of 0.09, outperforming SinGAN (0.11). This indicates that PatchGenCN more accurately captures the patch-level statistics of the source image.

Super-Resolution Performance

PatchGenCN demonstrates significant advantages in internal super-resolution. By learning the specific textures of a single image, it produces results with fewer artifacts and more realistic fine details compared to general-purpose or other internal learning models like ZSSR and DIP.

Super-resolution performance comparison across various models, including PSNR and NIQE metrics.

Conclusion

PatchGenCN provides a robust and theoretically grounded alternative to GANs for single-image learning. By explicitly modeling internal statistics through an energy-based framework, it achieves superior structural coherence and competitive realism. While the iterative nature of MCMC sampling results in slower inference compared to feed-forward generators, the model's simplicity, reduced parameter count (~0.99M), and versatility across tasks make it a significant advancement in deep internal learning.

Summary updated

Abstract

Exploiting internal statistics of a single natural image has long been recognized as a significant research paradigm where the goal is to learn the distribution of patches within the image without relying on external training data. Different from prior works that model such distributions implicitly with a top-down latent variable model (i.e., generator), in this work, we propose to explicitly represent the statistical distribution within a single natural image by using an energy-based generative framework, where a pyramid of energy functions parameterized by bottom-up deep neural networks, are used to capture the distributions of patches at different resolutions. Meanwhile, a coarse-to-fine sequential training and sampling strategy is presented to train the model efficiently. Besides learning to generate random samples from white noise, the model can learn in parallel to recover a real image from its incomplete version, which can improve the descriptive power of the learned models. The proposed model not only is simple and natural in that it does not require auxiliary models (e.g., discriminators) to assist the training, but also unifies internal statistics learning and image generation in a single framework. Qualitative results are presented on various image generation tasks, including super-resolution, image editing, harmonization, etc. The evaluation and user studies demonstrate the superior quality of our results.
BibTeX
@inproceedings{zheng2021patchwise,
  author = {Zilong Zheng and Jianwen Xie and Ping Li},
  title = {{Patchwise Generative ConvNet: Training Energy-Based Models From a Single Natural Image for Internal Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00298},
}
Figure