Prefrontal solution to the bias-variance tradeoff during reinforcement learning

Dongjae Kim, Jaeseung Jeong, Sang Wan Lee

Abstract

The goal of learning is to maximize future rewards by minimizing prediction errors. Evidence has shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.

Introduction: The Bias-Variance Dilemma in Reinforcement Learning

In both artificial intelligence and cognitive neuroscience, reinforcement learning (RL) is typically categorized into two distinct strategies. Model-Free (MF) learning relies on trial-and-error to form habits; it is computationally efficient and consistent (low variance) but slow to adapt to environmental shifts (high bias). Conversely, Model-Based (MB) learning involves building an internal map of the environment to plan future actions; it is highly flexible (low bias) but can be unreliable in noisy environments (high variance).

This "bias-variance tradeoff" implies that neither strategy is universally optimal. While previous theories suggested the brain "arbitrates" between these strategies based on their relative reliability, these models often fail when faced with "irreducible noise"—randomness in the environment that cannot be learned. Distinguishing between a genuine mistake and unavoidable noise is crucial for effective strategy selection.

Theoretical Framework: Adaptive Control and the Prediction Error Baseline

To address this, researchers proposed a new "Adaptive Control" hypothesis. The core idea is that the brain maintains a moving "prediction error baseline." By tracking the lower bound of errors, the brain can factor out background noise, allowing it to determine if a strategy is truly failing or simply encountering random variance.

The bias-variance tradeoff and the proposed adaptive control framework for reinforcement learning.

As shown in the figure above, different learning strategies (MF, MB, or a combination) result in varying levels of bias and variance. The adaptive control mechanism minimizes the lower bound of prediction error to optimize the learning strategy dynamically by factoring out irreducible noise.

Methods and Experimental Design

The study employed a multi-pronged approach combining theoretical modeling, behavioral experiments, and neuroimaging:

  • Behavioral Task: 82 participants performed a two-stage Markov decision task. The researchers manipulated "reward goals" (which outcomes were valuable) and "state-transition uncertainty" (the predictability of the task map).
  • Computational Modeling: A Dirichlet Process Gaussian Mixture Model (DPGMM) was used. Unlike fixed-threshold models, this non-parametric Bayesian approach allows the system to learn the distribution of errors online and adjust the baseline dynamically.
  • Neuroimaging: fMRI data from 22 participants were re-analyzed to identify brain regions tracking the prediction error baseline.

Key Results

Behavioral Evidence of Context Sensitivity

The "Adaptive Control" model significantly outperformed standard MB, MF, and previous arbitration models (Exceedance probability > 0.99). Participants demonstrated a clear "context sensitivity," shifting their choice optimality in response to environmental changes, such as shifts in reward goals.

Behavioral context sensitivity showing how reward goals influence decision-making strategies and normalized rewards.

Neural Mechanisms in the Prefrontal Cortex

The fMRI results validated the theoretical model, showing that the ventrolateral prefrontal cortex (vlPFC) and ventromedial prefrontal cortex (vmPFC) multiplex information. These regions represent both strategy reliability and the estimated noise baseline. Specifically, as the prediction error baseline increased—indicating more unavoidable noise—activity in the vlPFC rose, suggesting its role as a meta-level controller adjusting the balance between MB and MF learning.

Furthermore, the brain tracks dissociable baselines:

  • State Prediction Error (SPE) baseline: Related to the map of the world, reflected in the fusiform and lingual gyri.
  • Reward Prediction Error (RPE) baseline: Related to value, reflected in the pre-SMA, regulating functional connectivity between the vlPFC, posterior putamen, and vmPFC.

Discussion and Future Directions

This framework offers a robust explanation for how the brain navigates complex, noisy environments. By factoring out irreducible noise, the prefrontal cortex maintains the balance between flexible planning and efficient habit formation.

Beyond basic science, these findings have significant clinical implications. Psychiatric conditions like OCD or addiction may stem from a "stuck" baseline, leading to an over-reliance on model-free habits even when they are no longer effective. In the realm of Artificial Intelligence, this "context-aware" RL approach provides a blueprint for developing agents that can better handle real-world uncertainty, though scaling this to high-dimensional robotics remains a future engineering challenge.

Summary updated

Abstract

Abstract The goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning. One sentence summary A theoretical, behavioral, computational, and neural account of how the brain resolves the bias-variance tradeoff during reinforcement learning is described.
BibTeX
@misc{kim2021prefrontal,
  author = {Dongjae Kim and Jaeseung Jeong and Sang Wan Lee},
  title = {{Prefrontal solution to the bias-variance tradeoff during reinforcement learning}},
  year = {2021},
  doi = {10.2139/ssrn.3811830},
  howpublished = {SSRN},
}
Figure