Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation

Dotan Di Castro, Dmitry Volkinshtein, Ron MeirNeurIPS 2008 · OpenReview

This paper represents a significant bridge between the mathematical rigor of Reinforcement Learning (RL) and the biological realities of neuroscience. It provides a theoretical foundation for how the brain might actually implement "Actor-Critic" learning, specifically by modeling how dopamine modulates synaptic plasticity.

The Two-Time-Scale Problem

In Reinforcement Learning, Actor-Critic (AC) methods divide the agent into two distinct components: the Actor, which chooses actions, and the Critic, which evaluates those actions by estimating future rewards.

Historically, proving that these algorithms converge to an optimal solution required a "two-time-scale" assumption: the Critic had to learn at a much faster rate than the Actor. While mathematically convenient, this poses a major problem for neuroscientists. There is no biological evidence that different parts of the brain (such as the ventral and dorsal striatum) operate on vastly different learning schedules. Furthermore, direct policy methods often suffer from high variance, making learning slow and unstable.

Methods: Single Time Scales and Biological Realism

The authors introduce a batch-mode Temporal Difference (TD) based AC algorithm. Their primary theoretical contribution is a mathematical proof showing that this algorithm converges to a local optimum even when the Actor and Critic learn at the same time scale.

Key technical components of their approach include:

  • Regenerative Cycles: The algorithm updates based on "cycles" defined by visits to a recurrent state.
  • Likelihood Ratio Derivative: They utilize a specific gradient ascent method to adjust the Actor’s policy parameters.
  • The "f(x)" Flexibility: The authors' mathematical derivation includes an arbitrary function $f(x)$ in the gradient calculation. This clever addition allows the TD signal to be shifted to strictly positive values. This is crucial for biological realism, as biological neurons cannot have a "negative" firing rate; they can only modulate their activity from a baseline.

The authors then translate these abstract mathematical update rules into two neural models: a discrete-time McCulloch-Pitts model (resembling error-driven Hebbian learning) and a continuous-time Spiking Neuron model. In the latter, the update rule manifests as TD-modulated Spike-Timing-Dependent Plasticity (STDP), strongly mirroring phasic dopamine signals observed in the mammalian midbrain.

Experiments and Results

The paper uses synthetic environments to validate the theory, demonstrating that the single-time-scale approach is both mathematically sound and practically implementable in neural architectures.

1. The 36-State Maze The first environment is a grid-world with barriers where the agent receives a reward at a specific corner.

Neural network architecture, 36-state maze environment, and learning curves for different state representations.

The experiments highlighted that the quality of the Critic is heavily dependent on how the state space is encoded. Using a "one-hot" encoding (36 neurons for 36 states) performed significantly better than overlapping representations (12 neurons). Ultimately, the agent reached an average reward per stage of ~0.1, vastly outperforming a random agent's baseline of 0.005.

2. 2D Continuous Plane with Spiking Neurons The second task tested the spiking neuron model in a continuous 2D plane where an agent had to navigate to a central target using Newtonian dynamics.

Spiking neuron agent trajectories before and after learning, alongside the average reward curve.

The spiking neuron actor successfully learned to transition from an initial random walk to a direct, purposeful trajectory toward the target. The average reward ($\eta$) showed a clear, stable increase over time, proving the efficacy of the TD-modulated STDP approach.

Limitations and Future Work

While groundbreaking, the study has a few notable limitations:

  • Batch vs. Online Updates: The mathematical proof relies on batch-mode updates (waiting for a cycle to finish), whereas the neural implementations operate "online." A formal proof for the online setting was noted as future work.
  • Complexity and Scaling: The environments used are very small by modern standards. It remains unclear how this specific single-time-scale approach scales to high-dimensional problems.
  • Function Approximation: In the spiking neuron example, the Critic still relied on a linear function approximation rather than a fully spiking neural architecture, leaving a small gap in achieving a completely biologically plausible neural implementation.

Overall, this work successfully establishes that AC algorithms do not strictly require separated time scales, making them a much more plausible model for learning in the basal ganglia.

Summary updated

Abstract

Actor-critic algorithms for reinforcement learning are achieving renewed popularity due to their good convergence properties in situations where other approaches often fail (e.g., when function approximation is involved). Interestingly, there is growing evidence that actor-critic approaches based on phasic dopamine signals play a key role in biological learning through the cortical and basal ganglia. We derive a temporal difference based actor critic learning algorithm, for which convergence can be proved without assuming separate time scales for the actor and the critic. The approach is demonstrated by applying it to networks of spiking neurons. The established relation between phasic dopamine and the temporal difference signal lends support to the biological relevance of such algorithms.
BibTeX
@inproceedings{castro2008temporal,
  author = {Dotan Di Castro and Dmitry Volkinshtein and Ron Meir},
  title = {{Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=rkNcGFZ_WS},
}
Figure