On random weights for texture generation in one layer CNNS

Mihir Mongia, Kundan Kumar, Akram Erraqabi, Yoshua Bengio

TL;DR

This paper theoretically explains why one-layer Convolutional Neural Networks (CNNs) with random filters can effectively generate textures. They show that without non-linearities, random and pre-trained weights yield similar texture generation capabilities by preserving frequency magnitudes. However, introducing a ReLU non-linearity can lead to significantly less variable textures by restricting solutions to a unique input.

What Problem Does This Paper Address?

Recent research has demonstrated that even the initial layers of trained CNNs can generate realistic textures. Even more surprisingly, it's been shown that a single CNN layer with randomly initialized filters can also produce textures, albeit with less variability. This raises a fundamental question: why are these simple, one-layer random-weight CNNs so effective for texture generation? The authors aim to provide a theoretical explanation for this phenomenon, particularly focusing on the role of random weights and the impact of non-linearities.

Methods

The core of the paper involves a theoretical analysis of a simplified one-layer CNN model. They first analyze the model without any non-linearities, using circular convolution for mathematical tractability. They demonstrate that in this linear setting, the energy function used for texture generation (which aims to match the Gram matrices of the input and generated images) leads to solutions where frequency magnitudes are preserved, but phases are randomized. This explains why random weights can perform similarly to pre-trained weights when the number of filters is sufficiently large.

The paper then investigates the impact of adding a ReLU non-linearity. They show that in the one-dimensional case, the ReLU can drastically alter the solution space. While the linear model allows for infinite solutions that minimize the energy, the ReLU can, under certain conditions, constrain the solutions to a single, unique input (or its circular shifts).

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and uses synthetic examples (Figure 1) to illustrate concepts rather than relying on standard datasets for empirical evaluation.

Key Insights and Ideas

  • Random Weights Mimic Pre-trained Weights (in the Linear Case): The paper provides a theoretical basis for why random filters in a linear one-layer CNN can match the texture generation performance of pre-trained filters. The key is that the energy function, when applied to the Gram matrix, inherently preserves the magnitudes of frequency coefficients, regardless of whether the filters are random or learned.
  • Phase Randomization is Key: The effectiveness of random weights stems from their ability to randomize the phase of the frequency components while preserving their magnitudes. This is a crucial insight into texture synthesis.
  • ReLU Creates Uniqueness: The introduction of a ReLU non-linearity fundamentally changes the problem. Instead of infinite solutions, it can lead to a single, unique solution (or its circular shifts) that minimizes the energy. This is a significant departure from the linear case and explains why ReLU-based models can generate less variable textures.
  • Theoretical Foundation for Texture Generation: The work offers a theoretical framework for understanding texture generation using simple CNN architectures, bridging the gap between experimental observations and mathematical principles.

Key Results

  • Theoretically demonstrated that in a one-layer CNN without non-linearities, random weights and pre-trained weights yield similar texture generation capabilities by preserving frequency magnitudes.
  • Showed that the energy function used in texture generation inherently leads to phase randomization when applied to random filters.
  • Proved that in the one-dimensional case, the addition of a ReLU non-linearity can, under specific conditions, restrict the set of solutions that minimize the energy to a single unique input (or its circular shifts), leading to less variable outputs.
  • Illustrated with an example (Figure 1) how randomizing Fourier phases can generate textures, highlighting the potential for line artifacts, which the authors suggest might necessitate non-linearities.

Limitations and Future Work

  • Non-Convex Energy Function: The authors acknowledge that the energy function is non-convex with respect to input images, meaning generated images might get stuck in local minima, and the nature of these local minima is not fully understood.
  • Generality of ReLU Case: The full generality of the ReLU case has not been explored; the analysis primarily focuses on sufficient conditions where ReLU significantly differs from the linear model.
  • Multiple Layers: The current analysis is limited to one-layer CNNs. Extending this theoretical understanding to multi-layer architectures is a significant area for future work, although the authors suggest the current findings might offer intuition.
  • Dataset Independence: The theoretical nature of the paper means it doesn't rely on specific datasets for empirical validation, which could be a limitation for readers seeking practical benchmarks.

Summary updated

Abstract

Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures. More interestingly, it has also been experimentally shown that only one layer with random filters can also model textures although with less variability. In this paper we ask the question as to why one layer CNNs with random filters are so effective in generating textures? We theoretically show that one layer convolutional architectures (without a non-linearity) paired with the an energy function used in previous literature, can in fact preserve and modulate frequency coefficients in a manner so that random weights and pretrained weights will generate the same type of images. Based on the results of this analysis we question whether similar properties hold in the case where one uses one convolution layer with a non-linearity. We show that in the case of ReLu non-linearity there are situations where only one input will give the minimum possible energy whereas in the case of no nonlinearity, there are always infinite solutions that will give the minimum possible energy. Thus we can show that in certain situations adding a ReLu non-linearity generates less variable images.
BibTeX
@inproceedings{mongia2017random,
  author = {Mihir Mongia and Kundan Kumar and Akram Erraqabi and Yoshua Bengio},
  title = {{On random weights for texture generation in one layer CNNS}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2017.7952548},
}
Figure