Density Modeling of Images using a Generalized Normalization Transformation Johannes Ballé

Valero Laparra, Eero Simoncelli

Density estimation—learning the underlying probability distribution of data—is a fundamental challenge in unsupervised machine learning. For natural images, this task is notoriously difficult because pixels and their filter responses exhibit complex, non-Gaussian dependencies.

Standard linear methods like Principal Component Analysis (PCA) or Independent Component Analysis (ICA) can decorrelate data, but they fail to remove higher-order dependencies. For instance, if one edge in an image is strong, its neighboring edges are likely strong too. Previous nonlinear approaches, such as Radial Gaussianization (RG), work well for local clusters but fail as spatial distances between features increase. To address this, researchers have introduced Generalized Divisive Normalization (GDN), a biologically-inspired, flexible, and invertible transformation designed to "Gaussianize" natural image data.

The GDN Transformation

The GDN framework seeks to transform "messy" image statistics into a simple, independent Gaussian distribution. It achieves this through a two-stage process:

  1. Linear Transform: A standard matrix multiplication ($z = Hx$) initially rotates and scales the input data.
  2. Nonlinear Normalization: Each component $z_i$ is then divided by a "pooled activity measure" calculated from its neighbors: $$y_i = \frac{z_i}{(\beta_i + \sum_j \gamma_{ij} |z_j|^{\alpha_{ij}})^{\epsilon_i}}$$

Here, $\beta, \gamma, \alpha,$ and $\epsilon$ are learnable parameters. This structure generalizes "divisive normalization," a well-known model describing how neurons in the visual cortex regulate their gain.

Crucially, the authors developed a fixed-point iteration method to invert this transform. This invertibility is a superpower: it allows the model to function as a generative model. By sampling a random point in the Gaussian space and running it backward through the GDN, one can generate realistic-looking image patches. The parameters are optimized by minimizing negentropy, effectively measuring and reducing how far the transformed data deviates from a standard normal distribution. Furthermore, these transforms can be cascaded into deep architectures, with each layer progressively Gaussianizing the data.

Key Insights and Advantages

GDN succeeds where other methods fail because of its inherent flexibility. It acts as the "Goldilocks" of Gaussianization: it can capture "elliptical" dependencies (where data looks like a clustered cloud) at short distances, as well as "separable" dependencies at long distances. In contrast, ICA only handles the latter, and RG only handles the former.

Mutual information across spatial separations. GDN successfully minimizes dependencies across all distances, unlike ICA and RG.

The success of GDN also highlights a fascinating intersection of biological realism and statistical efficiency. The fact that a model derived from sensory biology performs so well at statistical density modeling suggests that biological visual systems may have evolved specifically to "Gaussianize" the visual world for highly efficient processing.

Experimental Validation and Results

The model was rigorously evaluated using standard image processing datasets, including the van Hateren dataset, the Kodak set, and BSDS300.

The results demonstrated significant improvements over existing methods:

  • Negentropy Reduction: GDN reduced negentropy by 2.43 nats per pixel, significantly outperforming both ICA-MG (2.04) and RG (2.11).
  • Distribution Matching: When analyzing the L2 norm of the transformed data, GDN perfectly matched the target theoretical distribution, proving its efficacy in true Gaussianization.
Distribution of the L2 norm of the transformed data. GDN closely matches the target chi distribution, indicating successful Gaussianization.
  • Visual Quality and Generative Power: Samples generated from the GDN model look remarkably like natural images, capturing "organic" structures and flat regions (like skies) that ICA-based models typically miss.
  • Denoising: When utilized as a prior for image denoising, GDN achieved a Peak Signal-to-Noise Ratio (PSNR) of 22.6 dB and a Structural Similarity Index (SSIM) of 0.78, outperforming standard Gaussian Scale Mixture (GSM) models.
  • Likelihood: On the BSDS300 dataset, GDN achieved a likelihood of 3.47 bits/pixel, matching the best-reported performance of significantly more complex models like RNADE.

Limitations and Future Directions

While highly effective, the GDN framework has limitations. Calculating the Jacobian determinant—which is required for the density model—is computationally expensive because it lacks a simple closed-form solution for the general GDN case. Additionally, to ensure the transform remains invertible and stable, specific constraints must be enforced on the exponents ($\alpha, \epsilon$) and weights ($\gamma$), which might limit the model's overall expressivity.

Future work will likely focus on exploring how these layers behave in much deeper architectures and whether they can be trained end-to-end for specific downstream tasks like image classification.

Summary updated

Abstract

We introduce a parametric nonlinear transformation that is well-suited for Gaussianizing data from natural images. The data are linearly transformed, and each component is then normalized by a pooled activity measure, computed by exponentiating a weighted sum of rectified and exponentiated components and a constant. We optimize the parameters of the full transformation (linear transform, exponents, weights, constant) over a database of natural images, directly minimizing the negentropy of the responses. The optimized transformation substantially Gaussianizes the data, achieving a significantly smaller mutual information between transformed components than alternative methods including ICA and radial Gaussianization. The transformation is differentiable and can be efficiently inverted, and thus induces a density model on images. We show that samples of this model are visually similar to samples of natural image patches. We demonstrate the use of the model as a prior probability density that can be used to remove additive noise. Finally, we show that the transformation can be cascaded, with each layer optimized using the same Gaussianization objective, thus offering an unsupervised method of optimizing a deep network architecture.
BibTeX
@inproceedings{laparra2016density,
  author = {Valero Laparra and Eero Simoncelli},
  title = {{Density Modeling of Images using a Generalized Normalization Transformation Johannes Ballé}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.06281},
}
Figure