Abstract
Modern computer vision architectures often decouple spatial and channel mixing. This study investigates whether learning spatial filters is necessary, finding that models with fixed, random spatial filters ("Chans-only") perform remarkably well. These models exhibit increased adversarial robustness and the ability to reconstruct images from scrambled pixels, suggesting that the core intelligence of neural networks lies in channel-wise feature combination rather than spatial scanning.
The Problem: Spatial vs. Channel Mixing
For decades, the 2D convolution has been the cornerstone of computer vision, simultaneously mixing information across space (neighboring pixels) and channels (feature maps). Recent architectures like Vision Transformers (ViTs) and ConvMixers have moved toward "isotropic" designs that explicitly decouple these operations. This separation raises a fundamental question: Which operation is doing the heavy lifting? This research investigates whether we actually need to learn spatial patterns, or if random spatial mixing provides a sufficient foundation for learned channel logic.
Methodology
The authors isolate spatial and channel mixing by utilizing Separable Convolutions. A standard convolution is split into:
- Depthwise Convolution (Spatial Mixing): Filters acting on each channel independently to capture spatial patterns.
- Pointwise Convolution (Channel Mixing): $1 \times 1$ convolutions that mix information across channels at a single pixel location.
The core experiment compares three model types: Full (all parameters learned), Chans (spatial filters frozen at random initialization), and Space (channel filters frozen).
Key Insights
Channel Mixing is the MVP
In almost every test, the "Chans-only" models performed nearly as well as fully trained models. In contrast, "Space-only" models failed significantly. This suggests that the network's "intelligence" resides primarily in how it combines features rather than how it scans the image.
The Power of Randomness and Spectral Coverage
Why do random spatial filters work? The authors propose Spectral Coverage: a large bank of random filters naturally covers a wide range of frequencies, effectively acting as a comprehensive set of sensors. As long as these sensors cover the frequency spectrum, the channel-mixing layers can learn to "tune in" to the necessary signals.
Natural Adversarial Defense
Fully learned models often over-optimize on high-frequency spatial details, making them vulnerable to adversarial noise. Because "Chans-only" models use fixed, random spatial filters, they are inherently more robust to attacks like FGSM and PGD. Smoothing these random filters further increased adversarial robustness by over 25% without hurting clean accuracy.
Decoding Chaos: Pixel Un-shuffling
Perhaps the most striking finding is that "Chans-only" models can solve the Pixel Un-shuffling task. Even when an image's pixels are randomly permuted—destroying all natural spatial context—these models can learn to reconstruct the original image almost as well as fully learned models.
Key Results
- Performance Gap: On ImageNet, the gap between a fully learned ConvMixer and a "Chans-only" version is remarkably small, despite fewer trainable parameters.
- Width Matters: As network width (number of channels) increases, the performance of "Chans-only" models converges toward fully learned models, supporting the idea that a sufficiently large bank of random filters will contain the necessary spatial features.
- Reconstruction: In pixel un-shuffling tasks, "Chans-only" models achieved PSNR values nearly identical to fully learned models, proving spatial learning isn't strictly necessary for reconstruction.
Limitations and Future Work
While compelling, the study focuses on isotropic models (ResNet and ConvMixer). Future work is needed to see if these findings translate to non-isotropic models or complex tasks like object detection. Additionally, while "Chans-only" models have fewer trainable parameters, they require the same inference computation as standard models.
Overall, this work suggests we may be over-engineering spatial filters and under-appreciating the power of channel-wise linear transformations.
Summary updated