On Random Weights for Texture Generation in One Layer Neural Networks

Mihir Mongia, Kundan Kumar, Akram Erraqabi, Yoshua Bengio

In computer vision, the success of neural networks is typically attributed to the optimization of weights through training. However, recent empirical evidence suggests that even a single-layer Convolutional Neural Network (CNN) with completely random weights can generate high-quality textures. This research, led by Mihir Mongia and colleagues, provides a theoretical framework to explain this phenomenon, focusing on the mathematical properties of the Gram matrix and the impact of non-linearities like ReLU.

The Role of Random Weights in Texture Synthesis

Texture generation involves creating new images that share the statistical properties of a target sample. While deep, pre-trained networks like VGG-19 are the standard for this task, studies have shown that a single layer with random filters can achieve comparable results. The researchers investigated whether this effectiveness is rooted in the preservation of specific image features regardless of weight training.

Target ridge texture used for analysis.

The study utilizes a simplified model: a single-layer CNN employing circular convolution. By minimizing an energy function—the difference between the Gram matrices of a generated image and a target texture—the network "learns" to produce a matching texture. The Gram matrix represents correlations between feature maps, which the authors analyze through the lens of the Discrete Fourier Transform (DFT).

Linear Models: Phase Randomization

In a linear one-layer CNN, the Gram matrix captures only the magnitude of the image's Fourier coefficients, entirely discarding phase information. This explains why random weights are effective: as long as the filters provide a "full rank" representation, they preserve the frequency magnitudes of the target.

The resulting generated textures maintain the "strength" of the original frequencies but scramble their spatial locations (phases). This leads to textures that look like the original but with shifted structural elements, such as the ridges seen in the target samples.

The ReLU Constraint: Breaking Invariance

The introduction of a Rectified Linear Unit (ReLU) non-linearity fundamentally changes the solution space. Unlike the linear case, which allows for infinite solutions via phase randomization, the ReLU model imposes significant constraints.

Comparison of generated textures under different constraints.

The authors prove that in a 1D signal processing context, as the number of random filters increases, the probability of the ReLU model having a unique solution (the original image itself, up to a circular shift) approaches 1. This theoretical finding explains why ReLU-based models often produce textures with less variability; the non-linearity breaks the phase-invariance of the linear layer, narrowing the pool of "valid" textures.

Key Theoretical Findings

  • Linear Equivalence: For any linear one-layer CNN, any circularly shifted version of the input image results in an identical Gram matrix (zero energy).
  • ReLU Uniqueness: With sufficient random filters, the ReLU model's global minimum is likely unique, effectively reconstructing the original signal rather than generating a diverse set of textures.
  • Filter Independence: In the linear case, random filters perform as well as trained ones because both preserve the same fundamental frequency magnitudes required for texture representation.

Conclusion and Future Directions

This work demystifies the surprising efficacy of random weights in simple CNNs. It highlights that while linear layers allow for texture diversity through phase randomization, non-linearities like ReLU act as powerful regularizers that can lead to signal reconstruction.

Future research is needed to extend these 1D proofs to the 2D image domain and to explore how these dynamics evolve in multi-layer architectures, where the interaction between successive non-linearities creates a more complex landscape for texture synthesis.

Summary updated

BibTeX
@inproceedings{mongia2017random,
  author = {Mihir Mongia and Kundan Kumar and Akram Erraqabi and Yoshua Bengio},
  title = {{On Random Weights for Texture Generation in One Layer Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2017.7952548},
}
Figure