TL;DR
Rehearsal Memory (RM) is a novel approach to long-sequence reasoning that mimics human cognitive processes to prevent "catastrophic forgetting" in neural networks. By combining a "history sampler" that identifies important past events with self-supervised "rehearsal" tasks, the model learns to prioritize and retain critical information within a fixed-size memory. This approach outperforms existing memory-augmented networks on complex text, video, and recommendation tasks.
What Problem Does This Paper Address?
Most AI reasoning tasks assume the model can always access the entire raw input (like a full book or a long video). However, for real-world applications like online recommendation systems or long-form video analysis, storing and re-processing massive sequences is computationally expensive and slow.
Memory-Augmented Neural Networks (MANNs) try to solve this by compressing inputs into a fixed-size "write-read" memory. But MANNs face two major hurdles:
- Gradual Forgetting: As new information comes in, older information is continuously overwritten and lost.
- Information Blindness: Standard MANNs treat all input data as equally important, often wasting limited memory slots on "noise" while forgetting the "signal" needed to answer a future query.
The Rehearsal Memory (RM) Framework
The authors propose the Rehearsal Memory (RM) framework, which introduces a segment-level memory machine and a dual-training strategy to combat these issues.
1. Rehearsal Memory Machine (RMM)
Instead of processing data token-by-token, the RMM breaks sequences into segments. It uses a Transformer encoder to capture local context and a gate-based update to write these features into memory slots.
2. How to Rehearse: Self-Supervised Tasks
To combat forgetting, the model "rehearses" past fragments using two self-supervised tasks:
- Recollection: The model takes a past fragment, masks 50% of its items, and must reconstruct them using only its current internal memory.
- Familiarity: The model must distinguish between a real fragment from its history and a "fake" one sampled from elsewhere.
3. What to Rehearse: The History Sampler
To solve the "importance" problem, the authors use a History Sampler. This acts as a "teacher" model that has temporary access to the full raw input during training. It calculates which fragments are most relevant to a specific query. The "student" (the RMM) then focuses its rehearsal efforts on these high-importance fragments, learning to prioritize task-relevant information.
Datasets and Key Insights
The model was evaluated across four diverse domains:
- bAbI: A synthetic reasoning benchmark for short-sequence QA.
- NarrativeQA: Long-sequence text QA based on book summaries (avg. 600+ tokens).
- ActivityNet-QA: Long-term video QA (avg. video length of 180 seconds).
- XLong: A lifelong sequence recommendation dataset consisting of Alibaba user click logs (sequence length of 1,000).
Key takeaways from the architecture include: * Segment-Level over Item-Level: Processing data in chunks (segments) allows the model to capture bidirectional context within each chunk, making the initial encoding much more robust than traditional RNN-based MANNs. * Privileged Training: The use of a "History Sampler" is a clever application of knowledge distillation. By letting a teacher model "peek" at the full data during training, the memory-constrained student learns better heuristics for what is worth keeping. * High Masking for Memory Strength: While models like BERT use a 15% masking ratio for pre-training, RM uses 50%. This high ratio forces the model to rely heavily on its long-term memory slots rather than just guessing based on the surrounding unmasked items in a fragment.
Key Results
RM consistently outperformed state-of-the-art memory models like the Differentiable Neural Computer (DNC) and the Compressive Transformer (CT).
- Accuracy: On the ActivityNet-QA video task, RM achieved 36.3% accuracy, surpassing the Compressive Transformer (35.4%) and significantly beating the DNC (30.3%).
- Recommendation: On the XLong dataset, RM achieved an AUC of 0.8817, outperforming specialized recommendation models like MIMN (0.8731).
- Efficiency: RM maintains a fixed memory size regardless of input length, providing $O(1)$ storage complexity, which is critical for scaling to "lifelong" sequences.
Limitations and Future Work
While RM shows significant improvements, the authors note that the "History Sampler" requires a separate training phase with access to raw inputs, which might not be feasible in all real-time learning scenarios. Additionally, while the model alleviates forgetting, the fixed memory size still acts as a fundamental bottleneck; determining the optimal number of memory slots ($K$) for different task complexities remains an open area for exploration. Future work may involve more dynamic memory allocation or exploring different rehearsal strategies beyond recollection and familiarity.
Summary updated