Learning to use working memory: a reinforcement learning gating model of rule acquisition in rats

Kevin Lloyd, Nadine Becker, Matthew W. Jones, Rafal BogaczFrontiers Comput. Neurosci 2012 · DOI

TL;DR

This paper introduces a reinforcement learning (RL) model that simulates how rats learn to use working memory to solve a rule-based maze task. The model successfully replicates rat behavior, including faster learning after rule reversals, and highlights how different RL algorithms can lead to distinct learning strategies and transfer of learning.

What Problem Does This Paper Address?

Learning to effectively use working memory is crucial for guiding behavior, especially in novel situations. A key challenge is determining what information to store and when to update it. Gating models offer a framework for this, conceptualizing working memory as a collection of past observations that are updated based on reinforcement learning signals. While these models have shown promise, their ability to precisely match animal behavior and the influence of different RL algorithms on their learning dynamics remain underexplored. This paper aims to bridge this gap by comparing two popular RL algorithms (Actor-Critic and SARSA) within a gating model to see how well they can replicate the rule acquisition and transfer learning observed in rats performing a spatial maze task.

Methods

The researchers developed a computational model that abstracts a spatial maze task. This model uses a gating architecture where a "gating agent" decides whether to update or maintain information in a single memory element, and a "motor agent" makes the actual movement decisions (e.g., turning left or right).

Two reinforcement learning algorithms were implemented for the gating agent:

  • Actor-Critic: This algorithm learns both state values (the "critic") and action preferences (the "actor") independently.
  • SARSA (State-Action-Reward-State-Action): This algorithm learns the values of state-action pairs directly.

Both algorithms use temporal difference (TD) learning with eligibility traces to update their internal value estimates. The models were trained on a simulated version of the rat task, and their parameters (learning rate, exploration temperature, eligibility trace decay) were fitted to match the rats' pre-reversal learning data using a Kullback-Leibler divergence minimization approach.

Datasets

The study used behavioral data collected from six adult male Long-Evans rats performing a spatial rule-learning maze task. This data served as the benchmark for evaluating the performance of the computational models. No new datasets were introduced.

Key Insights and Ideas

  • RL Algorithm Matters for Transfer Learning: The study found that the choice of RL algorithm significantly impacts how well the gating model replicates "positive transfer" of learning. SARSA showed faster learning after a rule reversal, mirroring the rats' behavior, while the standard Actor-Critic model did not. This suggests that the specific learning rules of the RL algorithm are critical for modeling phenomena like rapid adaptation to changing environments.
  • Multiple Strategies Emerge: Both the Actor-Critic and SARSA models, even with fixed parameters, learned multiple distinct strategies to solve the initial task. The most common strategies were "remember both" (maintaining information about both possible initial turns) and "remember one" (focusing on only one of the initial turns). This highlights the multi-agent nature of these models and suggests that individual differences in behavior might arise from these emergent strategy choices.
  • Task Difficulty and Working Memory: The difficulty of the task for both rats and models was influenced by the number of "distractor" observations between the initial cue and the final decision. This parameter (N) in the model can be interpreted as reflecting a baseline forgetting rate or limitations in working memory capacity, suggesting a link between task structure and cognitive constraints.
  • Emergent Timescales for Learning: The observed positive transfer in the SARSA model (and a modified Actor-Critic) was attributed to an "emergent effect" of different learning timescales for the motor and gating agents. The motor agent's actions are more immediately tied to reward, leading to faster updates, which helps preserve the learned memory policy during rule reversals.

Key Results

  • Both Actor-Critic and SARSA gating models successfully replicated the initial rule acquisition behavior of the rats, showing similar learning curves during the pre-reversal phase.
  • The SARSA gating model demonstrated significantly faster learning after a rule reversal, matching the rats' behavior (Figure 5B). The Actor-Critic model, without modification, did not show this positive transfer.
  • A modified Actor-Critic model (with bounded action values) also showed positive transfer, suggesting that the unbounded nature of its action values contributed to the lack of transfer in the original version.
  • Both models learned multiple strategies ("remember both" and "remember one"), with "remember one" being more common (Figure 5D).
  • The best-fit models required varying numbers of "distractor" observations (N) to match rat behavior, with N=4 for Actor-Critic and N=5 for SARSA.

Limitations and Future Work

  • Generalizability of RL Algorithms: The study emphasizes that the choice of RL algorithm is crucial and raises the question of which algorithms are biologically instantiated. Future work should explore other RL algorithms and their implications for working memory models.
  • Single Memory Element: The current model uses only a single memory element. Future work should extend the model to handle multiple memory elements to capture more complex working memory dynamics.
  • Neural Implementation: While the models provide behavioral insights, explicitly linking them to neural mechanisms remains a challenge. The authors suggest that future work should focus on relating model behavior to neural activity, particularly in tasks that allow for simultaneous monitoring of behavior and neural signals.
  • Beyond Simple RL: The paper notes that simple model-free RL algorithms have limitations in storing multiple policies, leading to maladaptive forgetting. Dayan's proposed uniform gating architecture, which can instantiate different rules, is suggested as a promising direction for future research to address these limitations and better model phenomena like serial reversal learning.
  • Spatial Encoding: The current model abstracts the maze into discrete points. Future work should explicitly encode spatial position to better relate model behavior to neuroscientific findings, such as observed spatial sequences in neuronal activity.

Summary updated

BibTeX
@article{lloyd2012learning,
  author = {Kevin Lloyd and Nadine Becker and Matthew W. Jones and Rafal Bogacz},
  title = {{Learning to use working memory: a reinforcement learning gating model of rule acquisition in rats}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2012},
  doi = {10.3389/fncom.2012.00087},
}

Loading related content...

Figure