TL;DR
This paper introduces a "hybrid" version of the Attention-Gated MEmory Tagging (AuGMEnT) reinforcement learning model, incorporating both leaky (short-term) and non-leaky (long-term) memory units. This modification allows the network to effectively handle hierarchical tasks requiring different timescales of memory, outperforming the original AuGMEnT on complex tasks like 12AX.
What Problem Does This Paper Address?
Many reinforcement learning (RL) agents struggle with tasks that require maintaining information over different timescales. For instance, some tasks demand remembering high-level context for extended periods, while simultaneously needing to quickly forget or update low-level details. The original AuGMEnT model, while biologically plausible, falters on such hierarchical tasks because its memory units are all "non-leaky," meaning they retain information indefinitely. This can lead to performance degradation as irrelevant old information interferes with learning new, relevant information. The authors aim to overcome this limitation by introducing a more flexible memory system.
Methods
The core innovation is the hybrid AuGMEnT network. This architecture modifies the original AuGMEnT by introducing two types of memory units:
- Non-leaky memory units: These function like the original AuGMEnT memory units, retaining information indefinitely (decay factor $\phi_j = 1$).
- Leaky memory units: These units have a controlled decay rate ($\phi_j < 1$), allowing them to forget information over time.
The network is trained using a reinforcement learning scheme, specifically the SARSA algorithm, where learning is driven by reward signals and employs synaptic tagging and attentional feedback mechanisms for credit assignment. The hybrid nature allows the network to learn to leverage both types of memory units as needed for a given task. The authors also introduce a modified weight function for action selection to improve prediction stability.
Datasets
The paper evaluates the hybrid AuGMEnT network on two cognitive tasks:
- Sequence Prediction Task: This task involves predicting the final letter of a sequence based on an initial cue, requiring the agent to maintain the cue while ignoring intermediate distractors. The difficulty scales with the length of the sequence (number of distractors).
- 12AX Task: A standard benchmark for testing working memory, this task requires identifying target pairs within a sequence of symbols. Its complexity arises from variable trial lengths and the need to distinguish between correct and incorrect target sequences, which can be seen as a generalization of a temporal XOR problem.
The paper does not introduce new datasets but uses established tasks from prior research.
Key Insights and Ideas
- Memory Timescale Matters: The success on complex, hierarchical tasks hinges on having memory units that can operate at different timescales. Indiscriminate long-term memory can be detrimental.
- Hybrid Memory is Flexible: Combining leaky and non-leaky memory units provides a more adaptable system. The network can learn to use non-leaky units for stable, high-level information and leaky units for rapidly changing or irrelevant details.
- Biological Plausibility Maintained: The hybrid approach retains the biological plausibility of the original AuGMEnT, building upon synaptic tagging and attentional feedback mechanisms.
- Task-Specific Memory Usage: The network learns to utilize its memory resources effectively, as evidenced by the weight matrices showing distinct patterns of activation for different memory unit types depending on the task.
Key Results
- 12AX Task Success: The original AuGMEnT failed to converge on the 12AX task, while the hybrid AuGMEnT achieved 100% convergence. A purely leaky control network also converged, suggesting the necessity of leaky dynamics for this task.
- Sequence Prediction Performance: Both the standard and hybrid AuGMEnT performed well on the sequence prediction task, outperforming a purely leaky control network. The hybrid model showed comparable or slightly better performance than the standard model.
- Efficiency: For the sequence prediction task, the hybrid model demonstrated good learning performance, converging in approximately 250 trials even with up to 10 distractors.
- Weight Analysis: Analysis of memory weights revealed that in the 12AX task, leaky units primarily stored digit cues, while non-leaky units handled inner loop information, demonstrating task-specific memory utilization.
Limitations and Future Work
- No Explicit Gating Mechanism: Unlike some other memory-augmented networks (e.g., HER, LSTM), the hybrid AuGMEnT lacks an explicit gating mechanism that learns when to store or recall information. The memory dynamics are driven by fixed decay coefficients.
- No Convergence Guarantees: The authors acknowledge that there are no theoretical convergence guarantees for their algorithm, especially given that the 12AX task is a Partially Observable Markov Decision Process (POMDP).
- Future Directions:
- Hierarchical Memory Structures: Extending the hybrid memory to multiple levels, each with distinct dynamics, could further enhance performance on complex hierarchical tasks.
- Learnable Decay/Gating: Making the decay coefficients or introducing gating mechanisms that are learnable would increase flexibility.
- Deep AuGMEnT: Exploring deep versions of the AuGMEnT network with additional hidden layers could be beneficial.
- Alternative Backpropagation: Investigating alternative backpropagation methods to maintain biological plausibility while enabling deeper architectures.
Summary updated