Thinking Deeply with Recurrence: Generalizing from Easy to Hard Sequential Reasoning Problems

Avi Schwarzschild, Arjun Gupta, Micah Goldblum, Tom Goldstein

Abstract

It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.

Introduction

In deep learning, "depth" is typically synonymous with stacking unique layers, each possessing its own set of parameters. The prevailing wisdom suggests that this architecture enables "layer specialization," where early layers learn simple features like edges, and deeper layers combine them into complex semantic objects. This paper investigates a fundamental question: Is this hierarchical behavior a result of having distinct parameters at each layer, or is it a result of sequential processing? By exploring recurrent neural networks (RNNs) that reuse the same filters across iterations, the authors challenge the assumption that overparameterization is necessary for high performance, suggesting that iterative computation is the true driver of depth-related benefits.

Methodology

The researchers introduce the concept of effective depth, which counts the total number of sequential operations performed, regardless of whether the weights are unique or shared. They compare standard feed-forward architectures (MLPs, ConvNets, and ResNets) against recurrent versions where a central module is applied multiple times.

Comparison between a recurrent model and its unrolled feed-forward equivalent, illustrating the concept of effective depth.

To analyze the internal dynamics of these "weight-shared" depths, several sophisticated techniques were employed:

  • Linear Separability Probes: Training small linear classifiers on intermediate representations to measure how class separability evolves at each iteration.
  • Centered Kernel Alignment (CKA): A metric used to quantify the similarity between the feature spaces of different models.
  • Feature Visualization: Optimization-based techniques (similar to DeepDream) to visualize what specific filters detect at different iterations.
  • Iteration-Specific Batch Normalization: A key technical detail where separate batch normalization statistics are maintained for each recurrence to handle varying feature distributions.

The hypothesis was tested across diverse benchmarks, including image classification (CIFAR-10, SVHN, EMNIST, and ImageNet) and a novel Maze Problem dataset. The maze task requires pixel-wise binary segmentation of the optimal path, a problem that inherently benefits from iterative "thinking."

Key Results and Insights

Accuracy Parity and Parameter Efficiency

Experimental results show that accuracy trends as a function of depth are remarkably similar between recurrent and feed-forward models. When a feed-forward model's performance plateaus at a certain depth, its recurrent counterpart typically does the same at the same effective depth. Notably, recurrent models match the performance of deep models while using significantly fewer parameters. For example, a recurrent ResNet with 3.56M parameters performed nearly as well as a standard ResNet-18 with 11.2M parameters on CIFAR-10.

Performance comparison across CIFAR-10, EMNIST, SVHN, and ImageNet benchmarks, showing how recurrent models (orange) closely track the performance of their feed-forward counterparts (blue) as effective depth increases.

Filter Recycling and Visual Hierarchy

A surprising finding is that the exact same convolutional filters can perform different roles depending on the iteration. In early iterations, a filter might detect simple textures; by the tenth iteration, that same filter (with the same weights) detects complex, abstract patterns. CKA analysis confirmed that an 8-layer feed-forward network is more similar to an 8-iteration recurrent network than to a 4-layer feed-forward network, suggesting that the computational path is more critical than parameter uniqueness.

Thinking Time in Reasoning Tasks

In the maze task, the authors found that harder (larger) mazes require more iterations to solve. This suggests that recurrence provides a way for a model to "think longer" about a problem without increasing its memory footprint. Performance continued to rise even after the "receptive field" of the network had already covered the entire input, indicating true iterative reasoning.

Conclusion

This work demonstrates that recurrent neural networks can emulate the performance and hierarchical feature extraction of deep feed-forward networks simply by increasing the number of iterations. While training very deep recurrent models remains an optimization challenge, the findings suggest that weight-sharing does not bottleneck the model's ability to learn complex hierarchies. Future work may explore "anytime prediction," where a recurrent model could stop iterating once it reaches a certain confidence level, potentially saving significant computation time during inference.

Summary updated

Abstract

It is widely believed that deep neural networks contain layer specialization, wherein neural networks extract hierarchical features representing edges and patterns in shallow layers and complete objects in deeper layers. Unlike common feed-forward models that have distinct filters at each layer, recurrent networks reuse the same parameters at various depths. In this work, we observe that recurrent models exhibit the same hierarchical behaviors and the same performance benefits with depth as feed-forward networks despite reusing the same filters at every recurrence. By training models of various feed-forward and recurrent architectures on several datasets for image classification as well as maze solving, we show that recurrent networks have the ability to closely emulate the behavior of non-recurrent deep models, often doing so with far fewer parameters.
BibTeX
@inproceedings{schwarzschild2022thinking,
  author = {Avi Schwarzschild and Arjun Gupta and Micah Goldblum and Tom Goldstein},
  title = {{Thinking Deeply with Recurrence: Generalizing from Easy to Hard Sequential Reasoning Problems}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=x8-eX9Uiv4},
}
Figure