TL;DR
This paper explores how working memory (WM) and episodic memory (EM) contribute to reinforcement learning (RL) agents tackling complex, non-Markovian tasks. By building a simplified RL agent with these memory systems and testing it on rodent-inspired tasks, the authors demonstrate the distinct roles of each memory type and identify scenarios where both are crucial for optimal performance.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents excel at learning optimal behaviors in environments where the current state fully dictates future outcomes (Markovian tasks). However, many real-world scenarios, including animal behavior, are non-Markovian, meaning the current sensory input isn't enough to make the best decision. To address this, researchers have proposed augmenting RL agents with internal memory systems, inspired by biological working memory (WM) and episodic memory (EM). While these approaches show promise, it's unclear how each memory system contributes individually and when they need to work together for increasingly complex decision-making tasks. This paper aims to disentangle these contributions by using simplified computational models and carefully designed tasks.
Methods
The authors employ a tabular actor-critic temporal difference learning architecture as their base RL agent. This agent learns by estimating the expected future reward for being in a given state (V-values) and for taking a specific action in a state (Q-values).
To incorporate memory, they introduce two key augmentations:
- Working Memory (WM): The state space is expanded to include a "hidden" WM element. This element can store the current sensory location state. The agent can update this WM element with an "update action" or use it to disambiguate states via a "cue retrieval" action. This allows the agent to retain information about the current context or cue within a single trial.
- Episodic Memory (EM): This system stores a history of the agent's most recently visited states. It's designed to be content-addressable and temporally indexed, allowing the agent to recall past episodes. The EM state can be set to a specific point in the past episode, enabling the agent to replay memories from that point.
These memory systems are integrated into the RL agent's state representation, allowing it to learn policies that depend on past experiences.
Datasets
The paper does not use traditional datasets. Instead, it introduces and evaluates its models on three distinct, rodent-inspired experimental tasks:
- Task A: Discrimination: A tactile cue at the start of a T-maze indicates which arm leads to a reward. This task requires the agent to remember the cue within a trial to make the correct choice.
- Task B: Alternation: In a T-maze, the agent must alternate its choices between two arms across trials to receive rewards. This task requires remembering the previous choice (inter-trial WM).
- Task C: Radial Maze: A more complex task involving multiple contexts, where the agent must learn an alternating rule within each context. This task requires maintaining memories for multiple contexts simultaneously (inter-task WM).
Key Insights and Ideas
- WM for within-trial and inter-trial context: WM is crucial for tasks where information needs to be retained during a single trial (like remembering a cue in Task A) or across consecutive trials (like remembering the previous choice in Task B).
- EM for complex contextual switching: For tasks with multiple distinct contexts (like the Radial Maze), where the rules change, EM becomes essential. It allows the agent to recall the specific history associated with each context, which WM alone struggles to manage when contexts are numerous or switch frequently.
- "Remember both" vs. "Remember one" strategies: In tasks like discrimination and alternation, agents can learn different strategies. "Remember both" involves storing all relevant information (e.g., both cue and choice), while "Remember one" relies on inferring the other piece of information. The paper shows how these strategies emerge and their implications for performance.
- The limitations of minimal RL: A basic RL agent without memory performs poorly on these non-Markovian tasks, often at chance level or worse. Even WM alone has limitations when tasks become very complex or involve many contexts.
Key Results
- Simple RL: The basic RL agent fails to learn any of the tasks effectively, performing at or below chance.
- RL with WM: The agent augmented with WM achieves optimal performance on the Discrimination (Task A) and Alternation (Task B) tasks. It learns to appropriately update and maintain WM representations for both intra-trial and inter-trial scenarios.
- RL with WM and EM: For the more complex Radial Maze task (Task C), WM alone is insufficient. The agent requires both WM and EM to achieve good performance, demonstrating the necessity of EM for managing multiple, distinct contexts.
- Strategy Analysis: The paper quantifies the emergence of different strategies ("remember both," "remember one," "remember other") in the WM agent, showing how they relate to task demands. For instance, in Task A, the agent learns to buffer the sensory cue, and in Task B, it buffers the previous choice.
Limitations and Future Work
- Ecological Validity: The authors acknowledge that the tasks, while inspired by rodent behavior, are simplified and not fully ecologically representative.
- Computational Complexity: While the proposed model is simpler than some existing RL approaches, the state space can still grow significantly with more complex memory requirements.
- Future Directions: The authors suggest that this work provides a foundation for testing predictions in experimental neuroscience and for developing more powerful learning algorithms. They highlight the need to explore how agents can adapt to non-stationary environments where rules change, and how to handle rule reversals more gracefully. The potential for biologically inspired cognitive control mechanisms for flexible decision-making is also mentioned as a future avenue.
Summary updated