Hierarchical Residuals Exploit Brain-Inspired Compositionality

Francisco M. López, Jochen Triesch

Abstract

Recent advances in deep learning have been driven by the introduction of residual connections, yet these typically operate on a local, layer-to-layer basis. In contrast, biological neural systems utilize long-range "shortcut" connectivity to integrate information across multiple hierarchical levels. This paper introduces Hierarchical Residual Networks (HiResNets), an architecture that mimics this biological strategy by incorporating skip connections across entire hierarchical blocks. Our results demonstrate that HiResNets achieve superior accuracy and faster convergence across CIFAR and Tiny-ImageNet benchmarks compared to standard and parameter-matched ResNet baselines.

1. Introduction

Residual Networks (ResNets) transformed the landscape of deep learning by enabling the training of extremely deep architectures through the use of skip connections. However, these connections are predominantly short-range, bypassing only a few layers at a time. This design choice contrasts sharply with the structural organization of the mammalian brain. Biological systems, from insects to primates, leverage long-range shortcuts that connect early sensory processing regions directly to multiple higher-level cortical stages.

This paper explores the implementation of "hierarchical compositionality" in artificial neural networks (ANNs). By allowing deeper layers to access raw or lightly processed data from much earlier stages, we investigate whether ANNs can benefit from the same "express lanes" found in nature.

2. The HiResNet Architecture

The fundamental innovation of HiResNets is the extension of the standard residual formula. In a conventional ResNet, the output $G(x)$ of a block is defined as $G(x) = F(x) + x$ (or a projection of $x$). In a HiResNet, the output of any given hierarchical block is the summation of its learned features and the projections from all preceding hierarchical blocks.

2.1 Hierarchical Projections

Integrating features from different hierarchical levels presents a challenge: spatial dimensions and channel depths vary significantly across the network. To resolve this, HiResNets utilize hierarchical projections consisting of:

  • Average Pooling: To downsample spatial dimensions (height and width) to match the current block.
  • 1x1 Convolutions: To align channel counts.
  • Batch Normalization: To stabilize the distribution of the integrated signals.

This mechanism ensures that deeper layers learn feature maps relative to the compressed representations of earlier stages, facilitating a more efficient optimization process.

3. Experimental Methodology

To evaluate the efficacy of HiResNets, we conducted experiments across three standard image classification benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet.

3.1 Baselines and Parameter Matching

We compared HiResNet variants against three primary architectures:

  1. Plain CNN: A deep convolutional network without skip connections.
  2. ResNet-18: The standard residual architecture.
  3. ResNeXt: A variant of ResNet utilizing grouped convolutions.

To ensure a fair comparison, we introduced "Adjusted" versions of these baselines. These models had their channel counts increased so that their total number of trainable parameters matched the HiResNet versions, accounting for the overhead introduced by the hierarchical projection layers.

4. Results

HiResNets consistently outperformed both the original and the parameter-matched "Adjusted" baselines across all tested datasets.

4.1 Accuracy Gains

The most significant improvements were observed on the Tiny-ImageNet dataset. The HiResNet version of the "Plain" network saw a dramatic increase in Top-1 accuracy, rising from 36.53% to 42.22%. The ResNet-18 architecture also benefited, with accuracy improving from 43.31% to 44.00%. These results suggest that the benefit of hierarchical residuals is not merely a byproduct of increased parameter count, but a result of the architectural topology itself.

4.2 Training Efficiency

Beyond final accuracy, HiResNets exhibited faster convergence. By providing the gradient with a more direct path to early layers—effectively shortening the "distance" between the error signal and the input—the models reached higher performance levels in fewer training epochs than their standard counterparts.

5. Discussion

Ablation studies were performed to identify which long-range connections contributed most to the performance gains. The "HiResNet-Out" variant, which specifically connects all previous blocks to the final stages of the network, performed nearly as well as the full HiResNet model. This indicates that shortcuts leading toward the output layers are particularly high-value, likely because they facilitate more effective gradient flow and allow the final classifier to leverage multi-scale features directly.

The findings suggest that hierarchical compositionality is a more effective use of a "parameter budget" than simply increasing depth or width. By forcing deeper layers to learn features relative to earlier representations, the network can build a more robust internal model of the data.

6. Conclusion and Future Work

HiResNets demonstrate that brain-inspired connectivity patterns can significantly enhance the performance and efficiency of artificial neural networks. While this study focused on ResNet-18 and CNN architectures, future research should explore the scalability of this approach in much larger models (e.g., ResNet-101) and its applicability to non-convolutional architectures like Vision Transformers (ViTs). Additionally, investigating how HiResNets handle incongruent features—where low-level textures conflict with high-level shapes—could provide further insights into the parallels between artificial and biological vision.

Summary updated

Abstract

We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels.HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy.We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning.A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.
BibTeX
@misc{lopez2025hierarchical,
  author = {Francisco M. López and Jochen Triesch},
  title = {{Hierarchical Residuals Exploit Brain-Inspired Compositionality}},
  year = {2025},
  eprint = {2502.16003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.16003},
}
Figure