Investigating Convolutional Neural Networks using Spatial Orderness

Rohan Ghosh, Anupam K. Gupta, Mehul MotaniWorkshops @ ICCV 2019 · OpenReview

TL;DR

This paper introduces spatial orderness, a novel statistical metric that quantifies how well 2D data adheres to spatial structures across different scales. By synthetically disrupting this order, the authors demonstrate that adding depth to Convolutional Neural Networks (CNNs) becomes counterproductive when data lacks spatial structure at higher scales, and they uncover a fascinating, counter-intuitive "bi-phasic" trend in feature map orderness during training.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) are exceptionally effective at processing images, audio, and graph data. This success is widely attributed to their built-in inductive biases: locality (pixels close to each other are related) and translation equivariance (a feature is the same no matter where it appears).

However, we still don't fully understand why and when these biases succeed based on the intrinsic characteristics of the data itself. Why do CNNs easily outperform fully connected networks even when global translational symmetry doesn't exist? And is more convolutional depth always better? This paper addresses these questions by shifting the focus from the network architecture to the structural properties of the data.

Methods

The authors propose a mathematical framework to measure spatial order, defined as the extent to which spatial proximity determines data value proximity (e.g., neighboring pixels having similar intensities).

  1. Spatial Orderness Metric ($so$): Inspired by a 2D spatial diffusion process, the authors look at triplets of pixels $(p, q, r)$ in a "two-hop" arrangement (where $q$ is a neighbor of $p$, and $r$ is a neighbor of $q$, but $r$ is not a neighbor of $p$). Under normal diffusion, the expected squared difference between $p$ and $r$ is exactly twice that of $p$ and $q$. If the data is completely random noise, this relationship becomes symmetric (a ratio of 1). The metric normalizes this ratio so that perfect spatial order equals 1 and random noise equals 0.
  2. Multi-Scale Extension: To measure order at different granularities, the authors downsample images by averaging non-overlapping $a \times a$ blocks and re-computing the metric.
  3. Random Block-Swapping: To test how CNNs handle disrupted data, they introduce a synthetic distortion technique. Images are divided into $k \times k$ blocks, and pairs of these blocks are randomly swapped for $N_s$ iterations, destroying spatial order at larger scales while preserving it within the blocks.

Datasets

The experiments were conducted using three standard image classification benchmarks:

  • MNIST
  • Fashion-MNIST
  • CIFAR-10

To evaluate their hypotheses, the authors generated 12 corrupted variants of these datasets using their random block-swapping technique (e.g., varying the number of swaps $N_s \in \{0, 10, 20, 30\}$).

Key Insights and Ideas

  • The Depth Limit on Disrupted Data: Convolutional layers exploit spatial order at multiple scales. If you destroy spatial order at higher scales via block-swapping, adding more convolutional layers ceases to provide any benefit and actually degrades test performance.
  • The Bi-Phasic Training Phenomenon: During the initial stages of training, the spatial orderness of feature maps across all layers undergoes a synchronized increase or "peaking" phase. Crucially, validation accuracy stagnates during this peak. Accuracy only begins to improve once the feature maps' spatial orderness starts to decline, showing that the network is successfully learning abstract, spatially de-correlated concepts.
  • Kernel Size Constraints: The authors prove theoretically and empirically that smaller convolutional kernels (like $3 \times 3$) naturally force the network to learn weights with high spatial orderness. As the kernel size expands toward the size of the input image (effectively turning the layer into a fully connected layer), this spatial orderness collapses.

Key Results

  • Quantifying Disruption: The spatial orderness metric successfully captures the effect of block-swapping. As shown in Figure 1, increasing the number of swaps ($N_s$) steadily drives the spatial orderness at higher scales down to zero.
Spatial orderness across scales with block-swapping
  • Performance Degradation: For original, unaltered data ($N_s = 0$), increasing network depth consistently lowers the test error rate. However, for highly swapped data ($N_s = 30$), adding layers beyond a certain point sharply increases the error rate (Figure 2).
Test error rates vs network depth for various block-swap intensities
  • Feature Map Dynamics: Figure 3 highlights the bi-phasic nature of training. Feature maps across layers peak in orderness within the first 10 epochs before declining as the validation accuracy curve finally takes off.
Evolution of feature map spatial orderness during training
  • Kernel Size vs. Orderness: When training on MNIST, increasing the first-layer kernel size from $K=3$ to $K=27$ cuts the spatial orderness of the learned weights nearly in half (Figure 4b), validating that small kernels are essential for maintaining spatial structure.
Impact of input disruption and kernel size on kernel spatial orderness

Limitations and Future Work

While the paper provides strong theoretical foundations and clear empirical proof, its scope is limited to relatively shallow CNNs (3 to 5 layers) evaluated on simple datasets (MNIST, CIFAR-10). The authors note that while the concept of spatial orderness can theoretically be extended to other domains—such as graph neural networks (GNNs) by redefining locality—they do not provide empirical validation for non-image domains.

Future work could explore how these spatial orderness dynamics behave in modern ultra-deep architectures (like ResNets) or vision transformers (ViTs), which lack the strict local inductive bias of CNNs.

Summary updated

Abstract

Convolutional Neural Networks (CNN) have been pivotal to the success of many state-of-the-art classification problems, in a wide variety of domains (for e.g. vision, speech, graphs and medical imaging). A commonality within those domains is the presence of hierarchical, spatially agglomerative local-to-global interactions within the data. For instance in natural images, neighboring pixels are more likely contain similar values than non-neighboring pixels which are further apart. To that end, we propose a statistical metric called spatial orderness, which quantifies the extent to which the input data (2D) obeys the underlying spatial ordering at various scales. In our experiments, we mainly find that adding convolutional layers to a CNN could be counterproductive for data bereft of spatial order at higher scales. Furthermore, we present a theoretical analysis (and empirical validation) of the spatial orderness of network weights, where we find that using smaller kernel sizes leads to kernels of greater spatial orderness and vice-versa.
BibTeX
@inproceedings{ghosh2019investigating,
  author = {Rohan Ghosh and Anupam K. Gupta and Mehul Motani},
  title = {{Investigating Convolutional Neural Networks using Spatial Orderness}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=dt8NdAwmfw},
}
Figure