TL;DR
This paper introduces "Linear Reinforcement Learning" (Linear RL), a novel computational model that explains how the brain can achieve flexible planning and decision-making by reusing past computations. It achieves this by approximating complex nonlinear optimization with a linear approach, incorporating a "default policy" that allows for biologically realistic biases and cognitive control, and offers a unified framework for understanding phenomena like grid cell representations and habits.
What Problem Does This Paper Address?
The brain's remarkable ability to plan and make decisions relies on reusing past computations. However, this reuse can also lead to inflexibilities like habits and compulsions. While existing models have attempted to explain this, they often struggle to fully account for both flexibility and the specific biases observed in behavior. Specifically, the challenge lies in creating a computationally tractable and neurally plausible model that can:
- Enable flexible replanning: The ability to adapt plans on the fly when goals or environments change.
- Explain graded biases: Why we sometimes exhibit subtle, graded tendencies (like Pavlovian biases) rather than rigid habits.
- Connect to neural representations: How brain structures like entorhinal grid cells might support this planning.
- Account for cognitive control: The effortful process of overriding automatic responses.
The core difficulty is the "interdependence of optimal actions" in sequential decision tasks, where the best choice now depends on the best choices at all future steps, making exhaustive planning computationally infeasible.
Methods
The authors introduce Linear Reinforcement Learning (Linear RL), a model inspired by control engineering. It addresses the planning problem by:
- Linear Approximation: Replacing the classic nonlinear, model-based optimization (which involves solving the Bellman equation recursively) with a linear approximation.
- Default Policy: Introducing a "default policy" ($\pi_d$) that represents a baseline set of action preferences. The model optimizes its decision policy ($\pi$) to be close to this default policy, incurring a "control cost" (quantified by KL divergence) for deviations.
- Default Representation (DR): This linear approximation allows for the computation of a stable, temporally abstracted map of future state expectancies, termed the Default Representation (DR). This DR is crucial because it's independent of specific goals and can be reused for planning under changing objectives.
- Matrix Operations: The DR is represented as a matrix that can be efficiently updated using matrix inversion identities, allowing for rapid replanning when environmental changes (like barriers) or goal revaluations occur.
This approach contrasts with the Successor Representation (SR), which is also based on state expectancies but is highly dependent on the learned policy, limiting its flexibility for replanning.
Datasets
The paper does not introduce new datasets. Instead, it uses simulations to evaluate its model on various tasks, including:
- A seven-level decision tree task.
- A maze environment.
- Simulations of Tolman's latent learning and detour tasks.
- A two-step Markov decision task with stochastic transitions.
- Simulations inspired by experimental data on Stroop effects and Pavlovian-instrumental transfer (PIT).
Key Insights and Ideas
- Unified Framework for Flexibility and Inflexibility: Linear RL provides a single computational framework that explains both flexible planning and the emergence of biases and habits. Inflexibility arises not from a failure of planning, but as a natural consequence of the "default policy" mechanism that enables efficient planning.
- Stable Cognitive Maps: The Default Representation (DR) acts as a stable, reusable map of long-term contingencies, independent of specific goals. This is a significant advantage over the Successor Representation (SR), which is policy-dependent.
- Componential Representation: The model suggests that brain representations like grid cells might encode basis functions for the DR, allowing for a componential understanding of cognitive maps, where different features (like barriers) can be added or modified.
- The "Cost" of Control: Cognitive control is framed as the cost incurred when deviating from the default policy. This provides a normative explanation for why overriding automatic responses can feel effortful.
- Biases as Efficient Planning Tools: Biases (like habits and Pavlovian effects) are not necessarily errors but can be seen as a consequence of a default policy that facilitates efficient planning by providing a baseline expectation.
Key Results
- Near-Optimal Performance: Linear RL achieved near-optimal average costs on a challenging seven-level decision tree task, outperforming several pruned model-based RL algorithms.
- Flexible Replanning: The model successfully explained flexible replanning in tasks like reward devaluation and detours, where the SR struggles.
- Neural Representation Insights: The DR's properties align with experimental findings on entorhinal grid cells, suggesting they might encode a basis for this stable map. The model also offers an explanation for border cells as representing additive components to the DR.
- Explaining Biases: Linear RL naturally accounts for phenomena like Stroop effects and Pavlovian-instrumental transfer by modeling the influence of a learned default policy.
- Stochastic Environments: The model can be extended to handle stochastic transitions, though it has limitations in certain complex scenarios.
Limitations and Future Work
- Approximation in Stochastic Environments: While the model can be extended to stochastic environments, it relies on an approximation that can be defeated in certain complex cases. Future work could explore more sophisticated approximation methods.
- Neural Implementation Details: The paper proposes that grid cells might encode eigenvectors of the DR, but the precise neural implementation of this decomposition and its dynamic updates remains an open question.
- Learning the Default Policy: While the paper demonstrates how an overtrained default policy can lead to habitual behavior, further research is needed to fully understand how the brain learns and adapts this default policy in various contexts.
- Componentiality and Nonlinearities: While the model offers a componential view of maps, fully translating and reusing these components (like border cells) might require additional nonlinearities.
- Defining Goals: The paper notes that the partitioning of states into terminal (goals) and nonterminal is somewhat flexible and could be further explored.
Summary updated