Standard Convolutional Neural Networks (CNNs) owe their immense success to a fundamental architectural assumption: spatial stationarity. By combining local receptive fields with global weight sharing, a standard convolutional layer assumes that a feature—whether it is a low-level edge or a high-level object part—maintains the same statistical meaning regardless of where it appears in an image.
However, this assumption breaks down when processing structured or globally aligned data. In tasks where absolute spatial coordinates carry critical semantic meaning, forcing a network to share weights across the entire input space can actively impede learning. Locally connected layers (also known as unshared local convolutions) solve this by decoupling local connectivity from weight sharing, allowing networks to learn region-specific features tailored to specific coordinates.
The Limits of Spatial Stationarity
To understand why locally connected layers work, we must first look at where global weight sharing fails. Consider an image of a human face that has been cropped and aligned. An eye will always appear in the upper-left and upper-right quadrants; a mouth will always appear near the bottom center.
Under a standard convolutional architecture, an eye-detecting filter is forced to scan the entire image, including the chin and forehead. This introduces two distinct inefficiencies:
- Statistical Dilution: The filter's weights must generalize across the distinct statistical properties of every region in the image, diluting its capacity to specialize in the fine-grained nuances of an eye.
- Spatial Disregard: The network loses explicit awareness of absolute position, meaning it cannot easily exploit the fact that an eye found on a chin is likely a false positive.
As demonstrated by Yaniv Taigman et al. in their landmark face verification system, DeepFace: Closing the Gap to Human-Level Performance in Face Verification, discarding global weight sharing in favor of locally connected layers allows different spatial regions to learn entirely unique feature extractors. This architectural shift ensures that the network optimizes its feature detection specifically for the expected anatomical structure at each coordinate.
This violation of spatial stationarity extends far beyond facial analysis:
- Genomics: In DNA sequence analysis, an identical nucleotide pattern can have completely different functional impacts depending on its exact location on a chromosome. Hyo-Jun Lee et al. utilized locally connected layers in deepGBLUP: joint deep learning networks and GBLUP framework for accurate genomic prediction of complex traits in Korean native cattle to capture these locus-specific marker effects, respecting the biological reality of physical positioning.
- High-Energy Physics: When classifying subatomic particle jets in calorimeter data, preprocessing steps typically fix the relative positions of key features. Kevin Bauer et al. demonstrated in Jet substructure classification in high-energy physics with deep neural networks that translation invariance becomes a distinct disadvantage in these structured classification domains, making locally connected architectures significantly more effective than standard convolutions.
Decoupling Locality from Parameter Reuse
Mechanically, a locally connected layer operates identically to a convolutional layer, with one crucial exception: the weights are untied.
If a convolutional layer applies a single $3 \times 3$ kernel across an entire $H \times W$ feature map, it requires only $3 \times 3 \times C_{in} \times C_{out}$ parameters. A locally connected layer, by contrast, allocates a completely independent $3 \times 3$ kernel to every single coordinate in the $H \times W$ grid, ballooning the parameter count by a factor of $H \times W$.
Standard Convolution:
[Kernel A] ──> Scans Position (1,1)
[Kernel A] ──> Scans Position (1,2)
[Kernel A] ──> Scans Position (1,3)
Locally Connected Layer:
[Kernel A] ──> Processes Position (1,1)
[Kernel B] ──> Processes Position (1,2)
[Kernel C] ──> Processes Position (1,3)
By isolating the benefits of local receptive fields from parameter reuse, this architecture proves a fundamental theoretical point. As eran malach showed in Computational Separation Between Convolutional and Fully-Connected Networks, the computational advantage of convolutions often stems from locality alone—the restriction of a neuron's field of view—rather than the strict enforcement of weight sharing.
Furthermore, weight sharing is not a hard requirement for achieving translation invariance. Jordan Ott et al. demonstrated in Learning in the Machine: To Share or Not to Share? that when networks with unshared local weights are trained with sufficient data augmentation, they can learn translationally invariant representations from scratch. Crucially, they do so while remaining highly sensitive to the global spatial arrangement of features—a property that standard CNNs often lose, as they tend to behave like bags of uncoordinated local features.
The Biological Imperative
Beyond engineering utility, locally connected layers bridge a massive gap between deep learning and neurobiology. Global weight sharing requires a mechanism to instantly copy and synchronize synaptic weights across millions of physically separated neurons—a feat that is biologically impossible.
In the primate visual cortex, neurons possess restricted receptive fields, but neurons processing different parts of the visual field develop distinct tuning distributions based on their specific localized input. Galen Pogoncheff et al. demonstrated this in Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture, showing that replacing weight-sharing convolutions with locally connected layers yields models that align far more accurately with real neural data from the primary visual cortex (V1).
When combined with biologically plausible learning rules like Spike-Timing-Dependent Plasticity (STDP) in Spiking Neural Networks (SNNs), locally connected layers allow individual neurons to naturally specialize in features unique to their spatial context. As shown by Daniel J. Saunders et al. in Locally Connected Spiking Neural Networks for Unsupervised Feature Learning, this localized competition enables rapid, unsupervised feature extraction that matches traditional engineering techniques while remaining highly compatible with neuromorphic hardware.
Taming the Parameter Explosion
The primary drawback of locally connected layers is their parameter footprint, which scales linearly with the spatial resolution of the feature maps. To combat this, researchers have developed several hybrid architectures that strike a balance between regional specialization and parameter efficiency.
Grid-Based and Local Weight Sharing
Instead of unsharing weights at the pixel level, a model can divide a feature map into a coarser grid of patches. Weights are shared globally within each patch, but each patch maintains a unique set of parameters.
Kaili Zhao proposed this middle ground in Deep Region and Multi-label Learning for Facial Action Unit Detection, demonstrating that grid-based local weight sharing captures regional statistics in structured images without triggering an unsustainable parameter explosion. Similarly, Yi Sun et al. successfully applied local weight sharing to higher layers of convolutional networks, allowing models to preserve spatial awareness where geometric features stabilize.
Dynamic Weight Generation
Rather than hardcoding unshared parameters into the network architecture, weights can be generated on the fly. Yuval Nirkin et al. introduced this concept in HyperSeg: Patch-Wise Hypernetwork for Real-Time Semantic Segmentation, using a hypernetwork to dynamically generate distinct weights for different image patches. This allows a single model to function as an ensemble of specialized regional networks tailored to the local context of a scene.
Involutions
Another alternative is to invert the traditional design of convolution entirely. Duo Li et al. proposed this in Involution: Inverting the Inherence of Convolution for Visual Recognition, creating an operator that is spatial-specific (different weights for different locations) but channel-agnostic (shared across channels). This inversion allows for wide receptive fields and location-specific modeling while drastically reducing the parameter redundancy typically found in channel-specific weights.
| Architecture | Receptive Field | Weights Across Space | Weights Across Channels |
|---|---|---|---|
| Standard Convolution | Local | Shared | Unshared |
| Locally Connected | Local | Unshared | Unshared |
| Grid-Based Sharing | Local | Shared within patch / Unshared across patches | Unshared |
| Involution | Local / Wide | Unshared | Shared |
Key Foundational Work
Learning to See Where and What: Training a Net to Make Saccades and Recognize Handwritten Characters (1992) — contributed 2 concept elements to this direction.
Multi-Layer Perceptrons with B-Spline Receptive Field Functions (1990) — contributed 1 concept elements to this direction.