In reinforcement learning, an agent learns to navigate an environment by maximizing a cumulative reward signal. To do this effectively, the agent must solve the "credit assignment problem": determining which specific actions in a long sequence were responsible for an eventual reward. Most modern solutions fall into the Actor-Critic framework, where an "actor" (the policy) learns what to do, and a "critic" (the value function) learns to predict the future rewards the actor will receive.
However, this framework is governed by a fundamental tension known as the bias-variance tradeoff. This tradeoff dictates the stability, speed, and ultimate success of the learning process.
The Two Extremes: Monte Carlo vs. Temporal Difference
To update a policy, we need an estimate of the "value" of an action. There are two classical ways to get this estimate:
- Monte Carlo (MC) Methods: The agent plays out an entire episode and uses the actual total reward as the learning signal. This is unbiased because it uses real data, but it has high variance. Because many random things can happen over a long trajectory, the signal is extremely noisy, making learning slow and unstable.
- Temporal Difference (TD) Learning: The agent doesn't wait for the end of the episode. Instead, it updates its current estimate based on the very next reward and its own future prediction (bootstrapping). This has low variance because it only looks one step ahead, but it is highly biased. If the agent's initial predictions are wrong, it will be learning from incorrect information, which can lead to divergence or sub-optimal strategies.
As noted by Aditya A. Ramesh and Jürgen Schmidhuber in Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning, managing this tension is the central challenge of credit assignment.
The Critic as a Variance Reducer
The primary role of the "critic" is to act as a baseline or a variance reduction tool. By providing a learned estimate of future rewards, the critic smooths out the noisy updates of the actor.
In High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman and colleagues introduced Generalized Advantage Estimation (GAE). GAE provides a mathematical "knob" ($\lambda$) that allows researchers to interpolate between the bias of TD and the variance of MC. This blending of timescales allows for stable learning in complex tasks like 3D locomotion.
Similarly, Théophane Weber and David Silver demonstrate in Credit Assignment Techniques in Stochastic Computation Graphs that learned critics can be generalized to any stochastic graph to lower gradient variance, though they warn that inaccuracies in the critic's training inevitably introduce bias into the policy updates.
The Hidden Cost of Low Variance
It is tempting to assume that the "best" algorithm is the one with the lowest variance. However, Wesley Chung and Nicolas Le Roux challenge this in Beyond Variance Reduction: Understanding the True Impact of Baselines on Policy Optimization. They prove that the minimum-variance baseline can actually be harmful, causing an agent to "commit" too early to a mediocre strategy. Sometimes, a bit of noise (variance) is necessary to keep the agent exploring and prevent it from converging to a sub-optimal solution.
Controlling the Bias
If the critic is inaccurate, the actor learns the wrong things. Several researchers have proposed methods to explicitly manage this "approximation bias":
- Maxmin Q-learning: Qingfeng Lan and Martha White show in Maxmin Q-learning: Controlling the Estimation Bias of Q-learning that by using multiple estimators, one can tune the bias to find a "Goldilocks zone" between overestimation and underestimation.
- Dual Actor-Critic: Albert Eaton Shaw and Lihong Li in Boosting the Actor with Dual Critic reframe the problem as a cooperative game. By using path regularization and multi-step bootstrapping, they stabilize the critic's predictions, reducing the harmful bias that often plagues deep RL.
- Non-Parametric Approaches: Jan Peters suggests in A Non-Parametric Approach to Dynamic Programming that using kernel density estimation can provide a more mathematically consistent way to model dynamics, bypassing some of the biases inherent in traditional function approximation.
Ultimately, the goal of this research direction is to move beyond simple "trial and error" and toward algorithms that can intelligently balance the clarity of real-world experience (Monte Carlo) with the efficiency of internal prediction (Temporal Difference).