The Uncanny Similarity of Recurrence and Depth

Avi Schwarzschild, Arjun Gupta, Amin Ghiasi, Micah Goldblum, Tom Goldstein

TL;DR

This paper demonstrates that recurrent neural networks—which reuse the same parameters over multiple iterations—can almost perfectly emulate the performance and hierarchical feature extraction of deep feed-forward networks. The findings suggest that the benefits of "depth" in AI are driven more by the number of computational steps (effective depth) than by the number of unique parameters or specialized layers.

The Problem: Layer Specialization vs. Parameter Reuse

In deep learning, the prevailing "layer specialization" hypothesis suggests that networks work because different layers do different jobs: shallow layers detect simple edges, while deeper layers assemble those edges into complex objects. This implies that having distinct, unique parameters for every layer is essential for performance.

However, recurrent neural networks (RNNs) break this rule by applying the same set of weights over and over. The authors ask: If a network reuses the exact same filters at every step, can it still achieve the same performance boosts we see in very deep models? Does it still develop a hierarchy of features? By answering these questions, the paper challenges our fundamental understanding of why "going deeper" actually works.

Methods: Unrolling Recurrence

The researchers compared three families of architectures—Multi-Layer Perceptrons (MLPs), Convolutional Networks (ConvNets), and Residual Networks (ResNets)—in two flavors:

  1. Feed-forward: Every layer has its own unique parameters.
  2. Recurrent: A single "module" (one or more layers) is repeated multiple times, reusing the same weights.
Recurrent vs. Feed-forward architecture

They introduced the concept of "Effective Depth"—the total number of sequential operations performed, regardless of whether the weights are shared. To ensure the recurrent models had a fair shake, they used separate Batch Normalization statistics for each iteration, acknowledging that feature statistics change as data passes through the same loop multiple times.

To look "under the hood," they used:

  • Linear Probes: Training small classifiers on intermediate layers to see how well the network "disentangles" categories at each step.
  • Feature Visualization: Using optimization (similar to DeepDream) to visualize what specific filters are looking for at different stages of the recurrence.

Datasets

The authors evaluated their models on several standard image classification benchmarks, including CIFAR-10, SVHN (Street View House Numbers), EMNIST (Extended MNIST), and ImageNet.

They also introduced a new Maze-Solving dataset. Mazes are a "sequential reasoning" task where the solution depends on global structure. They generated 180,000 mazes of varying sizes (9x9, 11x11, and 13x13) to test if recurrent models could handle tasks that traditionally require significant depth to resolve long-range dependencies.

Key Insights and Results

  • Depth is Computation, Not Just Parameters: The performance curves for recurrent and feed-forward models are nearly identical when plotted against "effective depth." This means a network doesn't necessarily need more parameters to get smarter; it just needs more time (iterations) to process the data.
Performance vs. Effective Depth across datasets
  • Performance Parity: On CIFAR-10, a recurrent residual network with only 12M parameters performed almost exactly as well as a feed-forward version with 31M parameters.
  • The Same Filter Can Do Two Jobs: Most surprisingly, the feature visualizations showed that the exact same convolutional filter can detect simple textures in the 2nd iteration but look for complex, abstract shapes (like eyes or faces) by the 16th iteration. The "specialization" is a result of the state evolving, not the weights changing.
  • Disentanglement is Incremental: Linear probes revealed that both model types "unfold" data into separable classes at almost the same rate per iteration.
  • Maze Solving: Both recurrent and feed-forward models showed a sharp "threshold" where they suddenly became able to solve mazes once they reached a certain effective depth. For large mazes, a minimum number of iterations was required regardless of whether the weights were shared.
  • Feature Similarity: Using Centered Kernel Alignment (CKA), the authors found that an 8-layer feed-forward net is more similar to its 8-iteration recurrent counterpart than it is to a 4-layer feed-forward net.

Limitations and Future Work

The authors acknowledge that their findings might not immediately generalize to every domain, such as medical imaging or complex object detection, where specialized filters might still hold an edge.

Furthermore, while they showed that recurrence can mimic depth, they didn't explore the "training stability" issues that often plague recurrent models (like vanishing gradients) at extreme depths. Future work could investigate the theoretical loss landscapes of these recurrent models to understand why they are so biased toward the same "flat minima" that make deep feed-forward networks generalize so well.

Summary updated

Abstract

It is widely believed that deep neural networks contain layer specialization, wherein networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits as depth despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.
BibTeX
@inproceedings{schwarzschild2022uncanny,
  author = {Avi Schwarzschild and Arjun Gupta and Amin Ghiasi and Micah Goldblum and Tom Goldstein},
  title = {{The Uncanny Similarity of Recurrence and Depth}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=x8-eX9Uiv4},
}
Figure