TL;DR
This paper proposes a novel mechanism within the basal ganglia, involving cholinergic tonically active interneurons (TANs), that allows for dynamic adjustment of learning rates based on uncertainty. This system enables more adaptive behavior in environments with fluctuating reward probabilities, outperforming fixed learning rate models.
What Problem Does This Paper Address?
Reinforcement learning (RL) is crucial for adapting behavior based on experience, but traditional RL models often struggle in environments where the relationship between actions and outcomes is uncertain or changes over time. While the basal ganglia are known to be involved in RL through dopamine-mediated reward prediction errors, it's unclear how they handle this uncertainty to optimize learning. Specifically, how can the basal ganglia dynamically adjust their learning rate to be robust to noise while also being flexible enough to adapt to changing circumstances? This paper addresses this gap by proposing a computational model where cholinergic signaling, specifically from TANs, plays a key role in encoding and responding to uncertainty.
Methods
The authors developed a computational model of the basal ganglia, extending a previously established neural network. Key to their approach is the inclusion of Tonically Active Neurons (TANs) and their interaction with medium spiny neurons (MSNs).
- Neural Network Model: An attractor network model of the basal ganglia was used, simulating Go and NoGo units (representing MSNs) and TANs. Learning was implemented using a contrastive Hebbian learning rule, modulated by dopamine signals.
- TANs and Uncertainty: TANs were modeled to have a burst-pause response during outcome feedback. The duration of this pause was manipulated, both exogenously (fixed duration) and endogenously (dynamically modulated by MSN population entropy).
- MSN Population Entropy: A measure of uncertainty was derived from the distribution of MSN firing rates, representing the degree of competition between different action representations.
- Feedback Control System: A key innovation was a feedback loop where MSN population entropy was used to dynamically adjust the TAN pause duration. This allowed the system to self-tune its learning rate.
- Algorithmic Models: To bridge the gap between biophysics and computation, the authors also developed an approximately Bayesian model and adapted an existing actor-critic model (OpAL) to incorporate the effects of TANs and uncertainty.
Datasets
Not discussed in the paper. The study relies on simulations of a computational model rather than empirical datasets.
Key Insights and Ideas
- TANs as Uncertainty Coders: The model suggests that TANs, through their pause duration, can act as a mechanism to encode and respond to uncertainty in the striatum.
- Dynamic Learning Rate Annealing: By modulating TAN pause duration based on MSN population entropy, the basal ganglia can dynamically adjust their learning rate, effectively "annealing" it based on the perceived uncertainty.
- Trade-off Between Flexibility and Robustness: The duration of the TAN pause creates a trade-off: longer pauses lead to more robust learning (less sensitive to noise) but slower adaptation to changes, while shorter pauses facilitate faster adaptation but can be more susceptible to spurious outcomes.
- Self-Tuning Mechanism: The feedback control system, where MSN entropy influences TAN pauses, allows the system to autonomously optimize its learning strategy for different environmental conditions.
- Algorithmic Equivalence: The neural network's behavior, particularly its response to uncertainty, is mirrored by both a Bayesian learner and an actor-critic model, suggesting a fundamental computational principle at play.
Key Results
- Improved Performance in Stochastic Environments: Networks with TANs generally showed higher asymptotic accuracy, especially in stochastic environments, compared to control networks without TANs.
- Adaptive Learning: Dynamically adjusting TAN pause duration based on MSN entropy led to superior performance across a range of reward schedules, mitigating the trade-off between flexibility and robustness seen with fixed TAN pauses. For instance, in an 85% reward environment, variable pause networks performed better than any fixed pause network.
- M1 Receptor Ablation Impairs Reversal Learning: Simulated M1 receptor ablation in the model mimicked empirical findings, showing impaired reversal learning while having less impact on initial acquisition.
- Post-Pause Burst Enhances Reversal: A simulated post-pause TAN burst improved asymptotic accuracy following reversal, suggesting a role in stabilizing newly learned contingencies.
Limitations and Future Work
- Source of Uncertainty Signal: While the model proposes MSN entropy as a source for modulating TAN pauses, the exact biophysical mechanism of how MSNs signal this uncertainty to TANs is still an open question, and direct in vivo evidence is lacking.
- Focus on M2 Receptors: The model primarily focused on M2 receptors for TAN modulation of MSN excitability, while acknowledging that TANs are influenced by other factors like dopamine and thalamic inputs.
- Simplified Bayesian Model: The Bayesian model used is an approximation and doesn't fully capture hierarchical inference.
- Future Work: The authors suggest testing their predictions experimentally, including investigating the correlation between MSN entropy and decision uncertainty, and the effect of M2 blockade on reversal learning. They also propose exploring more complex local striatal networks for TAN modulation and the role of TANs in integrating thalamic and dopaminergic signals.
Summary updated