Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks

Bokang Zhu, Richong Zhang, Dingkun Long, Yongyi MaoarXiv 2019 · arXiv

Abstract

Recurrent Neural Networks (RNNs) are cornerstone architectures for sequence modeling, yet their internal mechanisms for feature extraction and retention remain poorly understood. This paper investigates the intrinsic mathematical capabilities of various RNN architectures—including Vanilla RNNs, LSTMs, and GRUs—to perform simultaneous feature selection and memorization. By introducing the Flagged-1-Bit (F1B) test, we demonstrate a fundamental conflict between these two tasks. Our analysis reveals that while gated models resolve this conflict through dynamic gating, Vanilla RNNs require a higher state dimensionality to decouple loading and memorization into distinct neurons.

Introduction

Despite the success of RNNs in natural language processing and time-series analysis, they are often treated as "black boxes." A critical gap exists in understanding how these networks extract relevant temporal structures and maintain them against a stream of noise. This study formalizes these challenges as two distinct, often conflicting operations: feature selection (detecting structure) and feature memorization (preserving state).

The Flagged-1-Bit (F1B) Test

To isolate these capabilities, we propose the Flagged-1-Bit (F1B) test. In this synthetic task, a network receives a sequence of "bit pairs" consisting of an information bit ($x \in \{+1, -1\}$) and a flag bit. The flag bit remains at $-1$ except for a single random time step where it flips to $+1$. The network's objective is to output the value of the information bit corresponding to the $+1$ flag at the end of the sequence. Success requires the network to suppress noise, "load" the feature upon the flag's appearance, and "memorize" it until the sequence concludes.

The Selection-Memorization Conflict

Theoretical analysis identifies a fundamental incompatibility in sequence learning. Feature selection requires the network to amplify new inputs while suppressing the previous state. Conversely, memorization requires the network to suppress new (noisy) inputs to maintain the existing state.

Gated models, such as LSTMs and GRUs, resolve this conflict using dynamic gates. These mechanisms allow the network to switch its operational mode on the fly—from noise suppression to feature loading and back to memorization—using only a single state dimension ($K=1$).

Vanilla RNNs and the Dimensional Workaround

In contrast, a Vanilla RNN with static weights cannot satisfy these conflicting requirements within a single dimension. Mathematical proofs show that a Vanilla RNN fails the F1B test at $K=1$. However, it can succeed at $K=2$ by assigning distinct roles to its dimensions.

Vanilla RNN Dimensional Workaround

As shown in Figure 1, the network routes the flagged input through a "loading state" ($S^2$) and subsequently transfers it to a "memorization state" ($S^1$). This spatial separation allows the network to bypass the temporal conflict inherent in its fixed weights.

Empirical Validation

Monte-Carlo simulations confirm these theoretical findings. We tracked the internal state distributions of various models over time to observe how well they maintain signal separation.

State Distribution Simulations

In Figure 5, the top row illustrates a failure case where the distributions for positive and negative samples eventually overlap, indicating a loss of memory. The bottom row demonstrates successful memorization, where the signals remain perfectly separable even after many time steps, characteristic of gated models or higher-dimensional Vanilla RNNs.

Conclusion

This work clarifies the functional differences between gated and non-gated RNNs. While gating provides a more efficient, dynamic solution to the selection-memorization conflict, Vanilla RNNs are not intrinsically incapable; they simply require additional state dimensions to perform the same tasks. This suggests that the historical difficulty in training Vanilla RNNs may stem more from optimization challenges, such as vanishing gradients, rather than a lack of representational power. Future work will explore more complex "Flagged-k-Bit" tests to further probe these architectural limits.

Summary updated

Abstract

In this paper, we propose a test, called Flagged-1-Bit (F1B) test, to study the intrinsic capability of recurrent neural networks in sequence learning. Four different recurrent network models are studied both analytically and experimentally using this test. Our results suggest that in general there exists a conflict between feature selection and feature memorization in sequence learning. Such a conflict can be resolved either using a gating mechanism as in LSTM, or by increasing the state dimension as in Vanilla RNN. Gated models resolve this conflict by adaptively adjusting their state-update equations, whereas Vanilla RNN resolves this conflict by assigning different dimensions different tasks. Insights into feature selection and memorization in recurrent networks are given.
BibTeX
@misc{zhu2019understanding,
  author = {Bokang Zhu and Richong Zhang and Dingkun Long and Yongyi Mao},
  title = {{Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks}},
  year = {2019},
  eprint = {1903.00906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.00906},
}
Figure