Abstract
Recurrent Neural Networks (RNNs) are cornerstone architectures for sequence modeling, yet their internal mechanisms for feature extraction and retention remain poorly understood. This paper investigates the intrinsic mathematical capabilities of various RNN architectures—including Vanilla RNNs, LSTMs, and GRUs—to perform simultaneous feature selection and memorization. By introducing the Flagged-1-Bit (F1B) test, we demonstrate a fundamental conflict between these two tasks. Our analysis reveals that while gated models resolve this conflict through dynamic gating, Vanilla RNNs require a higher state dimensionality to decouple loading and memorization into distinct neurons.
Introduction
Despite the success of RNNs in natural language processing and time-series analysis, they are often treated as "black boxes." A critical gap exists in understanding how these networks extract relevant temporal structures and maintain them against a stream of noise. This study formalizes these challenges as two distinct, often conflicting operations: feature selection (detecting structure) and feature memorization (preserving state).
The Flagged-1-Bit (F1B) Test
To isolate these capabilities, we propose the Flagged-1-Bit (F1B) test. In this synthetic task, a network receives a sequence of "bit pairs" consisting of an information bit ($x \in \{+1, -1\}$) and a flag bit. The flag bit remains at $-1$ except for a single random time step where it flips to $+1$. The network's objective is to output the value of the information bit corresponding to the $+1$ flag at the end of the sequence. Success requires the network to suppress noise, "load" the feature upon the flag's appearance, and "memorize" it until the sequence concludes.
The Selection-Memorization Conflict
Theoretical analysis identifies a fundamental incompatibility in sequence learning. Feature selection requires the network to amplify new inputs while suppressing the previous state. Conversely, memorization requires the network to suppress new (noisy) inputs to maintain the existing state.
Gated models, such as LSTMs and GRUs, resolve this conflict using dynamic gates. These mechanisms allow the network to switch its operational mode on the fly—from noise suppression to feature loading and back to memorization—using only a single state dimension ($K=1$).
Vanilla RNNs and the Dimensional Workaround
In contrast, a Vanilla RNN with static weights cannot satisfy these conflicting requirements within a single dimension. Mathematical proofs show that a Vanilla RNN fails the F1B test at $K=1$. However, it can succeed at $K=2$ by assigning distinct roles to its dimensions.
As shown in Figure 1, the network routes the flagged input through a "loading state" ($S^2$) and subsequently transfers it to a "memorization state" ($S^1$). This spatial separation allows the network to bypass the temporal conflict inherent in its fixed weights.
Empirical Validation
Monte-Carlo simulations confirm these theoretical findings. We tracked the internal state distributions of various models over time to observe how well they maintain signal separation.
In Figure 5, the top row illustrates a failure case where the distributions for positive and negative samples eventually overlap, indicating a loss of memory. The bottom row demonstrates successful memorization, where the signals remain perfectly separable even after many time steps, characteristic of gated models or higher-dimensional Vanilla RNNs.
Conclusion
This work clarifies the functional differences between gated and non-gated RNNs. While gating provides a more efficient, dynamic solution to the selection-memorization conflict, Vanilla RNNs are not intrinsically incapable; they simply require additional state dimensions to perform the same tasks. This suggests that the historical difficulty in training Vanilla RNNs may stem more from optimization challenges, such as vanishing gradients, rather than a lack of representational power. Future work will explore more complex "Flagged-k-Bit" tests to further probe these architectural limits.
Summary updated