Introduction
Standard Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) units have become fundamental components in processing sequential data. However, most architectural innovations have focused on the internal structure of the individual cell. This paper, authored by researchers at Peking University, addresses a less explored area: the processing and transmission of information between adjacent units.
In biological systems, sensory stimuli are not directly committed to long-term memory. Instead, they undergo preliminary processing in a "working memory" buffer. Current RNN architectures often lack this nuanced pre-processing stage, which can lead to inefficiencies in discarding irrelevant information when new, more critical data arrives. To bridge this gap, the authors propose a bio-inspired framework based on the Atkinson-Shiffrin Modal Model of human memory.
The Proposed Framework
The core of the proposal is a three-stage architectural "wrapper" that can be applied to any standard recurrent unit, such as a basic RNN, LSTM, or GRU. This framework introduces a "look forward and forget" mechanism that allows the model to proactively manage its memory state.
The framework consists of three distinct stages:
- Working Memory ($h_t^{wm}$): This stage acts as a sensory buffer, taking the current input ($x_t$) and the previous long-term state ($h_{t-1}$) to create a preliminary "draft" of the current state.
- Forget Stage: Using the working memory state, the model calculates a "forget weight" ($F_t$). This weight is applied to the previous long-term store, allowing the model to "look forward" at new data to decide which past information is now obsolete.
- Long-term Store ($h_t$): The refined information is then passed into a standard recurrent unit to produce the final state for the current time step.
By decoupling the initial processing from the long-term storage, the framework allows for more sophisticated information filtering. The authors explored different forget functions, including a standard weighted sum and a parameter-efficient dot-product version that measures similarity between new inputs and old memories.
Experimental Evaluation
The framework was rigorously evaluated across several domains using six benchmark datasets:
- Text Classification: MR, CR, TREC, and Reuters.
- Image Classification: Sequential MNIST (pixels fed line-by-line).
- Language Modeling: Penn Tree Bank (PTB).
- Synthetic Task: A custom digital sequence task designed to test the model's ability to trigger memory clearing upon specific symbols.
Key Results and Discussion
The proposed framework consistently outperformed baseline models across all evaluated tasks. In text classification on the TREC dataset, the "F+RNN" (Framework + Basic RNN) achieved an accuracy of 92.8%, surpassing both standard RNN and LSTM baselines (both at 90.5%). For language modeling on the PTB dataset, the framework significantly reduced test perplexity from 89.335 to 84.899.
As shown in the training curves, the framework often demonstrates superior convergence or higher final accuracy compared to standard LSTMs. In synthetic tasks, the model successfully learned to drastically increase its "forget rate" when encountering specific trigger symbols, validating its ability to manage memory buffers effectively.
Conclusion
This research demonstrates that focusing on the transmission between units, rather than just the internal cell structure, can yield significant performance gains. By mimicking the multi-stage nature of human memory, the proposed framework provides a generalized way to enhance existing RNN architectures. While the "working memory" stage introduces some computational overhead due to additional parameter matrices, the improvements in complex, noisy sequence processing suggest it is a valuable addition for sophisticated deep learning models.
Summary updated