Convolutional Neural Networks on Randomized Data

Cristian IvanWorkshops @ CVPR 2019

TL;DR

This paper investigates the performance of Convolutional Neural Networks (CNNs) when trained on images with randomly permuted pixels. The study reveals that CNNs, which rely heavily on spatial hierarchies, suffer a significant performance drop compared to Multilayer Perceptrons (MLPs) when this structure is destroyed, highlighting the critical role of local correlations for CNNs.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) have revolutionized computer vision by effectively leveraging the inherent hierarchical and locally correlated structure of natural images. This structure allows CNNs to learn powerful representations with fewer parameters than traditional fully connected networks. However, a fundamental question arises: what happens to a CNN's performance when this crucial spatial structure is deliberately broken? Specifically, if the pixels within an image are randomly shuffled, can a CNN still learn effectively, or does its performance plummet? This paper aims to answer this by systematically evaluating CNNs on datasets where the spatial integrity of images has been compromised through pixel randomization.

Methods

The core of this study involves training and evaluating two types of neural networks: Convolutional Neural Networks (CNNs) and Multilayer Perceptrons (MLPs). The CNN architecture used is based on the VGG16 network, while the MLP comprises the last fully connected layers of this CNN, serving as a baseline. Both networks are trained using the Adam optimizer with a learning rate of 0.0001 and no data augmentation.

The primary experimental manipulation is pixel randomization. This is achieved through several methods:

  • Full Pixel Shuffling: The most straightforward approach, where all pixels in an image are randomly reordered.
  • Parametrized Patch Permutations: A more controlled method where images are divided into square patches, and these patches are then shuffled. This allows for varying degrees of randomization, from a single pixel (equivalent to full shuffling) to larger patch sizes that preserve more local structure.
  • Local Permutations: Pixels are swapped within a restricted neighborhood of a defined distance D. A distance of 0 means no permutation, while a distance equal to the image size is equivalent to full pixel shuffling.

Datasets

The experiments were conducted on the following standard image datasets:

  • MNIST: A dataset of handwritten digits.
  • Fashion-MNIST: A dataset of clothing images, noted for its higher complexity and inter-class similarities compared to MNIST.
  • CIFAR10: A dataset of 3-channel color images, featuring more diverse objects and less centered subjects than the other datasets.

Key Insights and Ideas

  • CNNs are highly sensitive to spatial structure: The most striking finding is the dramatic drop in CNN accuracy when trained on pixel-randomized images. This underscores that CNNs are not merely learning general patterns but are deeply reliant on the spatial arrangement of pixels.
  • MLPs are more robust to pixel randomization: In contrast to CNNs, MLPs (specifically the fully connected layers) show much less performance degradation when presented with permuted images. This suggests that while MLPs are less efficient for natural images, they are less dependent on the inherent spatial hierarchy.
  • Randomization destroys hierarchical features: The paper argues that pixel randomization destroys the hierarchical features that CNNs are designed to detect. While some residual class separation remains (preventing random guessing), the ability to learn complex, multi-level representations is severely hampered.
  • Dilated convolutions offer partial recovery: The introduction of dilated convolutional layers shows promise in recovering some performance on randomized data. These larger receptive fields can potentially capture longer-range correlations that might emerge even after pixel shuffling.
  • Color information is also leveraged by CNNs: Experiments with individual color channels (RGB) reveal that CNNs utilize color correlations for classification, as performance degrades more significantly when trained on single channels compared to the full RGB image, even with randomization.

Key Results

  • MNIST: CNN accuracy drops from 99.5% on natural images to 98.2% on permuted images. The MLP performance remains nearly identical for both natural and permuted images.
  • Fashion-MNIST: CNN accuracy falls from 94.3% to 89.6%. Again, the MLP shows minimal difference.
  • CIFAR10: This dataset shows the most pronounced effect. CNN accuracy drops from 88.9% to a mere 57.3% on permuted images, while the MLP remains largely unaffected.
  • Patch Permutations: Even moderate patch randomization (e.g., 16 slices per side, creating 2x2 pixel patches) significantly reduces CNN accuracy on CIFAR10.
  • Local Permutations: Increasing the randomization distance in local permutations leads to a steady decrease in CNN accuracy, eventually reaching levels similar to full pixel shuffling.

Limitations and Future Work

  • Limited exploration of "encrypted" data: While the paper touches upon homomorphic encryption preserving data structure, it doesn't deeply explore how CNNs might perform on truly encrypted data that might break spatial correlations more severely.
  • Architectural constraints of dilated convolutions: The authors note that the image size reduction inherent in dilated convolutions limits their application to deep architectures.
  • Future Work: The paper suggests designing networks that are more invariant or less sensitive to data permutations and encryption. It also highlights the potential for CNNs to analyze data where local correlations are not naturally present, such as in high-energy physics experiments, if architectures can be adapted to capture long-range correlations more effectively. The possibility of training on encrypted data and then reconstructing initial data from a few human-interpretable examples is also raised as an interesting avenue.

Summary updated

BibTeX
@inproceedings{ivan2019convolutional,
  author = {Cristian Ivan},
  title = {{Convolutional Neural Networks on Randomized Data}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure