Abstract
It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.
Introduction
The prevailing narrative in deep learning posits that "deeper is better" due to layer specialization. In this view, each successive layer in a feed-forward network employs a unique set of filters to extract increasingly complex features—progressing from simple edges to intricate semantic objects. This architectural design assumes that distinct parameters at each layer are necessary for building a feature hierarchy.
However, recurrent neural networks (RNNs) present a counter-example. In a recurrent architecture, data passes through the same layer multiple times, meaning the filters remain identical across every step of the computation. This research investigates whether the success of deep learning stems from having a vast number of unique parameters or if it is primarily a consequence of sequential processing, which we term "effective depth."
Methods
To investigate the relationship between recurrence and depth, we introduce the concept of effective depth: the total number of sequential operations an input undergoes, regardless of whether the weights in those steps are unique or shared.
We evaluated three families of architectures—Multi-Layer Perceptrons (MLPs), Convolutional Networks (ConvNets), and Residual Networks (ResNets)—by creating "twins" for each. A standard feed-forward version utilized unique weights for every layer, while a recurrent version repeated a single block to match the feed-forward model's effective depth. Performance was assessed on standard image classification datasets (CIFAR-10, SVHN, EMNIST, ImageNet) and a novel maze-solving benchmark requiring global reasoning.
Results
Our experiments reveal that recurrent models mimic the performance and hierarchical feature extraction of deep feed-forward networks with remarkable precision.
Performance Matching and Parameter Efficiency
As shown in the performance curves across various datasets, recurrent models track the accuracy of their feed-forward counterparts almost perfectly as effective depth increases. Notably, recurrent models achieve this while being significantly more parameter-efficient. For instance, on CIFAR-10, a recurrent ResNet reached 93.99% accuracy, nearly matching the 94.69% of a standard ResNet-18, despite using only about one-third of the parameters (3.5M vs. 11.2M).
Emergent Hierarchy with Shared Weights
Perhaps the most striking finding is that a hierarchy of features emerges even with shared weights. Through optimization-based feature visualization, we observed that the same filter extracts low-level textures in early iterations and complex semantic shapes in later iterations. This suggests that hierarchical representation is a property of sequential processing rather than parameter diversity. Linear probe analysis and Centered Kernel Alignment (CKA) further confirmed that the internal representations of recurrent models at iteration $N$ are more similar to feed-forward models at layer $N$ than to feed-forward models of different depths.
Discussion and Limitations
The "uncanny similarity" between recurrence and depth suggests that the benefits of depth in AI models are largely computational. While recurrence provides a path to high performance with fewer parameters, training very deep recurrent models remains challenging due to gradient stability issues, which we addressed using residual structures and batch normalization.
Future research should explore whether these findings generalize to other domains, such as natural language processing, and seek a deeper theoretical understanding of how shared weights manage to perform hierarchical feature extraction. The results here suggest that "effective depth" is a more critical metric for model capability than raw parameter count.
Summary updated