Understanding Deep Representations through Random Weights

Yao Shu, Man Zhu, Kun He, John E. Hopcroft, Pan Zhou

Abstract

Toward a deeper understanding on the inner work of deep neural networks, we investigate CNN (convolutional neural network) using DCN (deconvolutional network) and randomization technique, and gain new insights for the intrinsic property of this network architecture. For the random representations of an untrained CNN, we train the corresponding DCN to reconstruct the input images. Compared with the image inversion on pre-trained CNN, our training converges faster and the yielding network exhibits higher quality for image reconstruction. It indicates there is rich information encoded in the random features; the pre-trained CNN may discard information irrelevant for classification and encode relevant features in a way favorable for classification but harder for reconstruction. We further explore the property of the overall random CNN-DCN architecture. Surprisingly, images can be inverted with satisfactory quality. Extensive empirical evidence as well as theoretical analysis are provided.

Introduction

Deep neural networks, particularly Convolutional Neural Networks (CNNs), have achieved remarkable success in visual tasks, yet they are often regarded as "black boxes." A fundamental question remains: how much of this success is due to the learned weights versus the intrinsic properties of the convolutional architecture itself? This study isolates the architectural contribution by employing randomization—assigning fixed, random weights to a network—and using image reconstruction as a lens to evaluate the information captured by untrained structures.

Methodology

The researchers utilize standard architectures like VGG16 and AlexNet in two primary experimental setups:

  1. Random-CNN-trained-DCN (rwVGG): The CNN weights are fixed to random values drawn from various distributions (Gaussian, Uniform, Laplace). A corresponding Deconvolutional Network (DCN) is then trained on the ImageNet dataset to reconstruct input images from these random features.
  2. Purely Random CNN-DCN (rrVGG): Both the CNN and DCN use fixed random weights without any training, testing the "raw" signal recovery capability of the architecture.
Architecture of the Convolutional and Deconvolutional networks used for image inversion.

Evaluation is performed using the Structural Similarity Index (SSIM) and Pearson correlation coefficients on a validation subset of 5,000 ImageNet samples. A theoretical framework is also established to analyze the convergence of random CNN outputs as the number of channels increases.

Results and Analysis

The experiments yield several counter-intuitive insights regarding the nature of deep representations:

Training as a Reconstruction Bottleneck

Surprisingly, training a DCN on a random CNN converges significantly faster and achieves lower reconstruction loss than when paired with a pre-trained CNN. This suggests that training for classification acts as a discriminative filter, discarding structural information vital for reconstruction in favor of category separation.

Architectural Priors and Fidelity

Even in the purely random (rrVGG) setup, the network preserves geometric positions and contours, indicating that the convolutional structure acts as a strong "natural" prior for image data. Qualitatively, reconstructions from random VGG layers (rwVGG) are visually sharper and more accurate than those from pre-trained versions, particularly in deeper layers like Conv4 and Conv5.

Comparison of training loss and qualitative reconstruction fidelity between random and pre-trained networks.

Width vs. Depth

For random networks, channel width is more critical for reconstruction quality than depth. As signals move deeper, they become coarser due to pooling; however, increasing the number of random "views" (channels) helps recover lost information. Mathematically, the reconstruction error follows a $1/\sqrt{N}$ relationship with the number of filters $N$.

Conclusion

This work demonstrates that random features in a CNN are not merely noise but rich, high-dimensional encodings. The convolutional architecture itself provides a powerful structural prior that preserves significant image information even before training. Future research may explore how these insights apply to modern architectures like ResNets or Transformers and investigate hybrid training regimes that balance discriminative power with reconstruction fidelity.

Summary updated

Abstract

Toward a deeper understanding on the inner work of deep neural networks, we investigate CNN (convolutional neural network) using DCN (deconvolutional network) and randomization technique, and gain new insights for the intrinsic property of this network architecture. For the random representations of an untrained CNN, we train the corresponding DCN to reconstruct the input images. Compared with the image inversion on pre-trained CNN, our training converges faster and the yielding network exhibits higher quality for image reconstruction. It indicates there is rich information encoded in the random features; the pre-trained CNN may discard information irrelevant for classification and encode relevant features in a way favorable for classification but harder for reconstruction. We further explore the property of the overall random CNN-DCN architecture. Surprisingly, images can be inverted with satisfactory quality. Extensive empirical evidence as well as theoretical analysis are provided.
BibTeX
@misc{shu2017understanding,
  author = {Yao Shu and Man Zhu and Kun He and John E. Hopcroft and Pan Zhou},
  title = {{Understanding Deep Representations through Random Weights}},
  year = {2017},
  eprint = {1704.00330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.00330},
}
Figure