Abstract
We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels. HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy. We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning. A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.
Introduction
In the quest to build deeper and more powerful artificial neural networks (ANNs), researchers have long relied on residual connections (skip connections) to prevent vanishing gradients and training degradation. However, most modern architectures like ResNets primarily use "short-range" shortcuts that skip only one or two layers.
There is a notable discrepancy between AI and biology: the mammalian brain does not just rely on local connections. It features prominent "long-range" shortcuts from early subcortical areas (like the thalamus) directly to various stages of the cortical hierarchy. This paper explores whether replicating this multi-level connectivity in ANNs can improve how models learn complex, compositional features.
The HiResNet Architecture
HiResNets extend the standard residual formula. While a typical ResNet block computes $G(x) = F(x) + P(x)$, a HiResNet block integrates inputs from multiple previous hierarchical levels:
$$G(x_l | x_{l-1}, ..., x_0) = F(x_l) + P_1(x_{l-1}) + P_2(x_{l-2}) + ... + P_l(x_0)$$
To accommodate changes in spatial resolution and channel counts across different levels, the authors introduce Hierarchical Residual Projections. As shown in Figure 1, these projections consist of average pooling for downsampling, 1×1 convolutions for channel matching, and batch normalization for signal stabilization.
Experimental Results
The researchers evaluated HiResNets by modifying three baseline architectures—a "Plain" deep network, a ResNet-18, and a ResNeXt—across three standard image classification benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet.
HiResNets consistently outperformed their respective baselines. For instance, on Tiny-ImageNet, the HiResNet version of a Plain network improved Top-1 accuracy from 36.53% to 42.22%. Even the mature ResNet-18 baseline saw gains, reaching 89.46% on CIFAR-10 compared to the original 89.15%.
Importantly, the authors demonstrated that these gains are not merely due to increased parameter counts. "Adjusted" baseline models with increased channel counts to match HiResNet's parameters were still outperformed by the HiResNet variants, highlighting the efficiency of the hierarchical connectivity pattern.
Analysis and Insights
A key finding is that HiResNets facilitate hierarchical compositionality. Deeper layers do not learn features from scratch; instead, they learn feature maps relative to the compressed representations provided by the long-range skip connections.
Furthermore, these connections act as "gradient highways," providing direct paths from early layers to the end of the network. This allows gradients to flow more easily during backpropagation, leading to faster convergence. Ablation studies revealed that connections linking the very first layers to the final blocks (the most "subcortical-like" shortcuts) provided the most significant performance boosts.
Conclusion
HiResNets demonstrate that brain-inspired long-range connectivity can significantly improve the performance and training efficiency of deep convolutional networks. While the current study focused on ResNet-18 depths, future work will explore the scalability of these hierarchical residuals to much deeper architectures and large-scale Vision Transformers.
Summary updated