Abstract
Recurrent Neural Networks (RNNs) like Gated Recurrent Units (GRUs) are often viewed as "black boxes" due to their recursive complexity. This study investigates how a vanilla GRU-based sequence-to-sequence model performs precise token-positioning tasks without attention mechanisms. By employing a three-stage identification strategy—hypothesis generation, importance filtering, and causal manipulation—we uncover a specialized internal choreography. We identify four distinct neuron types—storing, counting, triggering, and outputting—that work in concert to manage temporal and content-based information. Our results demonstrate high task accuracy and provide causal evidence for these functional roles, offering a new perspective on the interpretability of recurrent architectures.
Introduction
The interpretability of RNNs remains a significant challenge in deep learning. While modern architectures like Transformers rely on explicit attention mechanisms for alignment, vanilla Seq2Seq models can still execute complex tasks such as "token-positioning" (e.g., placing a specific word at a designated index). This research aims to bridge the gap between performance and interpretability by answering how memory-constrained RNNs track time and store specific information without external alignment modules.
Methods
The authors utilized a vanilla GRU-based encoder-decoder model (no attention, no bidirectional layers) trained on the Gutenberg and Lyric datasets. The task required the model to output a specific token at a target position $T$ based on control signals. To identify functional neurons, a rigorous three-stage strategy was developed:
- Hypothesis & Candidate Generation: Support Vector Machines (SVMs) and Recursive Feature Elimination (RFE) were used to find groups of neurons whose values correlated with specific tasks (like storing a token).
- Filtering: Integrated Gradients were applied to quantify the actual importance of these candidate neurons to the final output, filtering out those that were merely correlated but not causal.
- Verification via Manipulation: To prove causality, researchers manually manipulated neuron values. For example, zeroing out "storing" neurons or resetting "counting" neurons to a previous state to observe if the model’s behavior changed predictably.
Key Insights: The Division of Labor
The most striking discovery is the specialized "division of labor" among neurons. The paper identifies four functional roles that enable precise sequence generation:
- Storing Neurons: These act as the model's content memory, holding the identity of the target token from the moment it is seen in the encoder until it is needed in the decoder.
- Counting Neurons: These serve as the model's internal clock. Their values change linearly at each time step, allowing the network to track time without explicit memory of previous states.
- Triggering Neurons: These act as a bridge, shifting values dramatically just before the target position to signal the model to stop counting and start outputting.
- Outputting Neurons: These interact directly with the final output layer to boost the probability of the specific stored token.
The interaction is precise: triggering neurons essentially "unlock" the storing neurons, allowing the token information to flow into the outputting neurons at the exact moment determined by the counting neurons.
Results and Causal Verification
The vanilla GRU model achieved high token-positioning accuracy: 98.5% on the Gutenberg dataset and 98.1% on the Lyric dataset. Causal manipulation experiments provided definitive proof of neuron functionality:
- Content Dependency: When "storing" neurons were zeroed out, accuracy plummeted to 10.7%, proving their essential role in token retention.
- Temporal Control: By replacing "counting" neuron values with those from $k$ steps prior, researchers could delay the output by exactly $k$ steps with nearly 99% accuracy. This confirms that these neurons directly control the model's internal timing mechanism.
Conclusion
This study successfully "exposes" the internal mechanics of vanilla GRUs, showing that they do not operate as monolithic black boxes but rather as coordinated systems of specialized units. While the inference phase is now better understood, the emergence of these roles during the training phase remains a mystery. Future work may explore whether these mechanisms scale to larger vocabularies or if attention-based models like Transformers employ similar internal counting logic.
Summary updated