Introduction
Standard Recurrent Neural Network Language Models (RNNLMs) compress the entire history of a conversation or text into a single hidden vector. This vector is expected to represent everything: immediate syntax, long-term context, and overarching "topic" or "meta-information." This puts significant "stress" on a single hidden state, especially when the topic is ambiguous or changes. While standard RNNs have an implicit representation of topics, they often struggle to learn them effectively without explicit guidance.
The Active Memory Network (AMN) Architecture
To address this limitation, the authors propose the Active Memory Network (AMN). Instead of one hidden state, it uses $K$ parallel, time-dependent memory cells (memcells).
The architecture consists of several key components:
- The Controller: A separate RNN that processes the input and previous controller state to determine the relevance of specific memory cells.
- Attention Mechanism: The controller generates a "soft-attention" vector, assigning weights to each of the $K$ memory cells.
- Response Output: The final representation for word prediction is a weighted sum of all memory cells.
By splitting the hidden state into multiple cells, the model can dedicate different cells to different "eigen-topics," making the learning process more efficient and modular. This allows the network to store multiple specialized representations and choose the most relevant one at each time-step.
Training and Regularization
Training such a complex architecture is challenging because the model tends toward "greedy" behavior, relying on only one cell. To mitigate this, the authors introduced two key techniques:
- Attention-weight Annealing: A temperature parameter that starts high to force equal usage of all cells, gradually lowering to allow specialization.
- Implicit Target Loss (ITL): A novel regularization term that encourages individual memory cells to specialize by pushing them closer to the final aggregated output, effectively de-coupling them and discouraging redundant co-adaptation.
Experimental Results
The AMN was evaluated on two major benchmarks: the Penn Tree Bank (PTB) and the BBC Multi-Genre Broadcast (MGB) dataset. The MGB dataset is particularly challenging, containing 700M words across seven distinct genres like News, Comedy, and Documentary.
On the PTB dataset, the AMN achieved an evaluation perplexity of 91, a significant improvement over standard GRU (114) or LSTM (117) baselines. In the MGB task, the AMN achieved a Word Error Rate (WER) of 27.0%, outperforming the GRU baseline (27.2%) and standard 3-gram models (28.5%). Notably, the AMN naturally learned to associate specific memory cells with specific genres without explicit labels.
Conclusion
Active Memory Networks provide a robust framework for language modeling by explicitly managing internal representations through attention. By reducing representation stress and allowing for unsupervised specialization, AMNs achieve superior performance in both perplexity and speech recognition accuracy compared to traditional recurrent architectures.
Summary updated