Deep reinforcement learning (DRL) holds immense promise for continuous control tasks, ranging from complex locomotion to real-world robotic manipulation. However, scaling value-based and actor-critic methods to continuous action spaces introduces severe mathematical and optimization instabilities. Unlike discrete environments where an agent selects from a finite set of actions, continuous domains require navigating infinite action possibilities. This shifts the burden onto neural network function approximators, which frequently suffer from compounding value estimation errors, high gradient variance, and inefficient exploration.
To achieve state-of-the-art performance, recent research focuses on stabilizing these core algorithmic components. By mitigating estimation biases, structuring exploration, and integrating structural priors, these methods bridge the gap between theoretical reinforcement learning and stable, real-world continuous control.
Taming Value and Gradient Estimation Biases
In actor-critic frameworks, the actor relies entirely on the critic’s value landscape to improve its policy. If the critic systematically overestimates or underestimates rewards, the actor updates its policy in a suboptimal direction, leading to catastrophic divergence or premature convergence to local optima.
To address this, He et al. introduce the WD3 algorithm in Taming the Estimation Bias in Deep Reinforcement Learning. The core intuition is that value estimation bias is rarely uniform; algorithms alternate between overestimation and underestimation. By adaptively weighting two distinct critics, the authors balance these biases dynamically, preventing the policy from chasing phantom rewards or prematurely abandoning promising paths.
Value estimation errors are also exacerbated by rapid policy updates. When a policy changes too quickly, the target values used to train the critic become moving targets, causing massive approximation errors. Gao et al. mitigate this in Error controlled actor-critic by introducing an Error Controlled Actor-Critic (ECAC) framework. By explicitly limiting the divergence between consecutive policies, the authors bound the value function's approximation errors, ensuring stable value estimation across continuous control tasks.
Standard Actor-Critic:
[Actor Policy] ──(Rapid Updates)──> [Unstable Critic Landscape] ──> Catastrophic Divergence
Error-Controlled Actor-Critic (ECAC):
[Actor Policy] ──(Bounded Divergence)──> [Stable Critic Landscape] ──> Consistent Convergence
Bias also arises from the use of historical data. In off-policy learning, agents utilize data collected by older versions of the policy. The mismatch between the historical policy and the current policy introduces off-policy bias. Saglam et al. resolve this in Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach by introducing a parameter-free policy similarity metric. This metric corrects for data collected by older policies, allowing actor-critic methods to utilize off-policy data efficiently without getting trapped in local optima.
Optimization dynamics themselves can introduce instability. Within the Generalized Policy Iteration (GPI) framework, policy evaluation and policy improvement can pull network gradients in conflicting directions. Xiao et al. tackle this in CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration by enforcing gradient consistency between the evaluation and improvement steps, ensuring that policy updates directly align with value maximization. Furthermore, high gradient variance can severely slow down convergence. Zhao et al. address this in Stochastic Variance Reduction for Deep Q-learning by combining Stochastic Variance Reduced Gradient (SVRG) techniques with the Adam optimizer, dramatically reducing gradient variance and accelerating sample efficiency.
Structuring Exploration in Infinite Spaces
Standard exploration strategies, such as adding isotropic Gaussian noise to actions, are highly inefficient in high-dimensional continuous spaces. Because they sample blindly, agents either over-explore suboptimal regions or fail to discover narrow paths to high rewards, leading to premature convergence.
One way to overcome this is to separate the mechanisms of exploration and exploitation entirely. Yang et al. propose this architecture in Efficient Reinforcement Learning via Decoupling Exploration and Utilization. By employing separate optimistic (exploration-focused) and pessimistic (utilization-focused) actors, the authors ensure that aggressive exploration does not corrupt the stable exploitation policy, maximizing sample efficiency without sacrificing safety.
Alternatively, exploration can be managed through evolutionary mechanisms. Karush Suri introduces ESAC in Off-Policy Evolutionary Reinforcement Learning with Maximum Mutations, combining the global exploration capabilities of evolutionary strategies with the sample efficiency of Soft Actor-Critic (SAC). By introducing an Automatic Mutation Tuning (AMT) mechanism, this approach abstracts exploration from exploitation, mitigating hyperparameter sensitivity and enhancing robustness across complex locomotion tasks.
Exploration can also be dynamically scaled based on the agent's performance. Yan et al. introduce Goal Achievement Guided Exploration (GAGE) in Goal Achievement Guided Exploration: Mitigating Premature Convergence in Learning Robot Control. Instead of uniform exploration, GAGE dynamically scales exploration based on how effectively the robot is achieving its target goals, preventing the agent from getting stuck in suboptimal behaviors.
For high-dimensional continuous spaces, Aghabozorgi et al. develop Practical $\epsilon$-Exploring Thompson Sampling (PETS) in Practical $\epsilon$-Exploring Thompson Sampling for Reinforcement Learning with Continuous Controls. By combining Thompson Sampling with Langevin Monte Carlo for posterior sampling over parallel Markov chains, the authors provide a mathematically grounded exploration strategy that stabilizes leading DRL algorithms and prevents premature convergence.
Integrating Models, Priors, and Hierarchies
Learning entirely from scratch (pure model-free RL) in continuous control is sample-inefficient and dangerous for physical hardware. Integrating structural priors, imperfect analytical models, or hierarchical abstractions provides a stabilizing "safety rail" or roadmap for the learning process.
De Lellis et al. demonstrate this integration in Control-Tutored Reinforcement Learning: Towards the Integration of Data-Driven and Model-Based Control. Their framework, Control-Tutored Reinforcement Learning (CTQL), injects a pre-existing, albeit imperfect, model-based controller as a "tutor" into the Q-learning process. This hybrid approach guides the agent's early exploration, significantly reducing steady-state errors and sample requirements.
CTQL Framework:
[Imperfect Model-Based Tutor] ──(Guidance)──┐
▼
[Environment State] ─────────────────> [Q-Learning Agent] ──> Optimized Action
Similarly, Koç et al. propose bayesILC in Optimizing the Execution of Dynamic Robot Movements With Learning Control. For highly dynamic tasks like table tennis, the authors combine adaptive model estimation with a cautious control strategy that explicitly accounts for model uncertainty, ensuring stable tracking performance in physical deployments.
When transitioning between simulated environments and the real world, dynamics mismatches can destabilize policies. Niu et al. address this in H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps. By explicitly modeling the dynamics gap between offline datasets and online simulators, the authors allow agents to safely transition from offline pre-training to online refinement, maintaining balance and achieving target velocities in real-world robotic tasks.
Structural stability can also be achieved by redefining the reward formulation or task hierarchy:
- Reward Formulations: Adamczyk et al. introduce ASAC in Average-Reward Soft Actor-Critic, optimizing for average rewards rather than traditionally discounted rewards. This provides a more robust mathematical formulation for long-horizon locomotion tasks where discounting distorts long-term value.
- Policy Hierarchies: Chen et al. propose a Hierarchical Ensemble DRL (HED) framework in Ensemble Reinforcement Learning in Continuous Spaces - A Hierarchical Multi-Step Approach for Policy Training, which trains an ensemble of base learners via a multi-step high-level policy to ensure convergence stability in complex continuous spaces.
- Subtask Trees: For long-horizon planning, Wang et al. develop a two-stage framework in Learning Top-K Subtask Planning Tree Based on Discriminative Representation Pretraining for Decision-making that pre-trains discriminative representations for subtasks and builds a dynamic planning tree, allowing agents to handle complex environments robustly.
Finally, handling partial observability—a major source of instability in real-world robotics—is addressed by Nguyen et al. in Leveraging Mutual Information for Asymmetric Learning under Partial Observability. The authors leverage privileged state information during training via mutual information maximization, enabling the agent to learn robust information-seeking behaviors that persist even when privileged data is removed during execution.
Key Foundational Work
WD3: Taming the Estimation Bias in Deep Reinforcement Learning (2020) — contributed 1 concept elements to this direction.
Stochastic Variance Reduction for Deep Q-learning (2019) — contributed 1 concept elements to this direction.
Optimizing the Execution of Dynamic Robot Movements With Learning Control (2019) — contributed 1 concept elements to this direction.