Self-Instantiated Recurrent Units with Dynamic Soft Recursion

Aston Zhang, Yi Tay, Yikang Shen, Alvin Chan, Shuai ZhangNeurIPS 2021 · OpenReview

Introduction

Standard Recurrent Neural Networks (RNNs), such as LSTMs and GRUs, process data in a linear "chain" structure. While this works well for simple sequences, much of our world—from the syntax of human language and computer code to the structure of music and logic—is fundamentally hierarchical or recursive.

Existing models usually attempt to capture this hierarchy in one of two ways: they either require a pre-defined "tree" (like a grammar parser) or they use fixed, multi-layered stacks. The former is rigid and requires extra annotated data, while the latter lacks the flexibility to spend more "thinking time" on a difficult part of a sequence and less on an easy one.

To bridge this gap, researchers have developed the Self-Instantiated Recurrent Unit (Self-IRU), a new type of neural network layer that uses "recursion within recurrence." By allowing a recurrent unit's gates to be controlled by smaller instances of itself, the model learns to dynamically adjust its computational depth for every single step in a sequence, achieving state-of-the-art results on complex logical and algorithmic tasks.

The Architecture of Self-IRU

The core innovation of this paper is the Self-IRU. Unlike a standard RNN where gates (like the "forget gate") are simple linear transformations, the Self-IRU's gates are determined by invoking another instance of a Self-IRU.

Internal architecture of a Self-IRU cell showing recursive gating mechanisms.

Key components of this architecture include:

  • Self-Instantiation: The forget gate and output gate at a certain recursion depth are calculated by invoking a version of the model at a lower depth. This creates a nested, tree-like structure of computation within a single time step.
  • Dynamic Soft Recursion: The model does not just recurse blindly. It uses "soft depth gates" (denoted as $\alpha$ and $\beta$) that take values between 0 and 1. These gates act as interpolators, allowing the model to decide how much to rely on the deep recursive call versus a simpler "base transformation."
  • Base Transformations: At the bottom of the recursion (depth 0), the model uses standard components like LSTMs or linear layers. The Self-IRU is "agnostic" to these, meaning it can be wrapped around various existing architectures.

When unrolled over time, this mechanism allows the network to maintain a dynamic, tree-like recursive structure that adapts at every step of the sequence.

Tree-like recursive structure of Self-IRU unrolled across consecutive time steps.

Key Insights and Capabilities

Recursion as a Parameter Expander Instead of just adding more layers, the Self-IRU dynamically expands its parameters through self-invocation. This allows it to be remarkably parameter-efficient. For example, on sequential image classification (CIFAR-10), it achieved 73.01% accuracy with only 0.9M parameters, nearly matching the performance of much larger models like TrellisNet (73.42%) which uses 8.0M parameters—effectively matching performance with only 1/8th of the parameters.

Data-Driven Complexity Analysis of the "soft gates" showed that the model actually learns the complexity of the data it processes. For simple sequences like grayscale images (MNIST), the gates remained constant. However, for complex color images (CIFAR-10) or rhythmic music, the gates fluctuated aggressively, showing the model was actively "tuning" its recursion depth per time step based on the immediate difficulty of the data.

No Syntax Needed One of the most surprising findings was that the Self-IRU outperformed models that had access to ground-truth syntactic trees on logical inference tasks. This suggests that its "soft" learned recursion is more effective and flexible than hard-coded grammatical rules.

Performance Across Modalities

The authors evaluated the Self-IRU across a diverse range of modalities to prove its flexibility, yielding impressive results:

  • Logical Inference: Achieved state-of-the-art accuracy (e.g., 88.0% on the most complex operations) on the Bowman et al. (2014) dataset, outperforming the previous best model (ON-LSTM) by a wide margin (+12%).
  • Music & Code: It achieved the best performance (lowest negative log-likelihood) on the Nottingham and Piano Midi datasets and set new performance bars for the Django code generation task (translating language to Python code).
  • Algorithmic Tasks: Successfully tackled sorting sequences of integers and Tree Traversal tasks.

Limitations and Future Work

While the paper demonstrates the power of the Self-IRU, a few areas remain for exploration. First, recursive calls can introduce computational overhead compared to simple linear layers, though the authors note that the maximum recursion depth used in experiments was quite small (usually 1 or 2). Second, the model's performance still depends on the choice of the "base transformation" (e.g., LSTM vs. Linear), and the optimal choice varies by task. Finally, the authors suggest that developing a better theoretical or biological understanding of why these specific recursive patterns emerge could be a fruitful area for future research.

Summary updated

Abstract

While standard recurrent neural networks explicitly impose a chain structure on different forms of data, they do not have an explicit bias towards recursive self-instantiation where the extent of recursion is dynamic. Given diverse and even growing data modalities (e.g., logic, algorithmic input and output, music, code, images, and language) that can be expressed in sequences and may benefit from more architectural flexibility, we propose the self-instantiated recurrent unit (Self-IRU) with a novel inductive bias towards dynamic soft recursion. On one hand, theSelf-IRU is characterized by recursive self-instantiation via its gating functions, i.e., gating mechanisms of the Self-IRU are controlled by instances of the Self-IRU itself, which are repeatedly invoked in a recursive fashion. On the other hand, the extent of the Self-IRU recursion is controlled by gates whose values are between 0 and 1 and may vary across the temporal dimension of sequences, enabling dynamic soft recursion depth at each time step. The architectural flexibility and effectiveness of our proposed approach are demonstrated across multiple data modalities. For example, the Self-IRU achieves state-of-the-art performance on the logical inference dataset [Bowman et al., 2014] even when comparing with competitive models that have access to ground-truth syntactic information.
BibTeX
@inproceedings{zhang2021self,
  author = {Aston Zhang and Yi Tay and Yikang Shen and Alvin Chan and Shuai Zhang},
  title = {{Self-Instantiated Recurrent Units with Dynamic Soft Recursion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=7Da3azsjjlh},
}
Figure