Abstract: This paper discusses parameter-based exploration methods for reinforcement learning. Parameter-based methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Parameter-based exploration unifies reinforcement learning and black-box optimization, and has several advantages over action perturbation. We review two recent parameter-exploring algorithms: Natural Evolution Strategies and Policy Gradients with Parameter-Based Exploration. Both outperform state-of-the-art algorithms in several complex high-dimensional tasks commonly found in robot control. Furthermore, we describe how a novel exploration method, State-Dependent Exploration, can modify existing algorithms to mimic exploration in parameter space.
Introduction
In continuous Reinforcement Learning (RL), an agent must balance exploiting known good behaviors with exploring new ones. The default exploration strategy in Policy Gradient (PG) methods (like REINFORCE) is "action-perturbing" exploration: injecting Gaussian noise into the action output at every single time step.
This creates several major problems for practical applications:
- High Variance: Sampling noise at every step injects massive variance into the gradient estimates, significantly slowing down convergence.
- Credit Assignment: If an agent takes 1,000 noisy actions in an episode and succeeds, it is incredibly difficult to determine which specific random perturbations actually contributed to the success.
- Hardware Damage: For physical robots, adding independent noise at every time step results in erratic, high-frequency "shaky" movements that can damage motors and fail to execute smooth tasks.
The authors tackle this by advocating for an alternative: exploring in the parameter space rather than the action space.
Methodology
The paper contrasts standard action-perturbing methods with parameter-based methods, highlighting three specific algorithms within the "Direct RL" (Policy Gradient) paradigm.
- PGPE (Policy Gradients with Parameter-Based Exploration): Instead of a probabilistic policy that outputs noisy actions, PGPE maintains a probability distribution over the parameters (weights) of a deterministic policy. At the start of an episode, it samples one set of parameters and uses them for the entire rollout. It then updates the distribution's mean and variance based on the episode's total reward.
- NES (Natural Evolution Strategies): An algorithm from the black-box optimization world. Like PGPE, it maintains a Gaussian distribution over parameters, but it updates them using the natural gradient, which accounts for correlations between parameters and handles ill-shaped fitness landscapes more effectively.
- SDE (State-Dependent Exploration): A novel "converter" method that allows standard PG algorithms to mimic parameter exploration. Instead of adding independent noise at each step, SDE uses a pseudo-random function that takes the state as input to generate noise. This ensures that if the agent encounters the exact same state twice in an episode, it applies the exact same exploration noise, resulting in smooth, consistent trajectories.
Experimental Evaluation
The authors evaluated the algorithms on four simulated physics benchmarks implemented via the Open Dynamics Engine and the PyBrain library:
- Extended Pole Balancing: A classic control task, made harder by starting with completely random policies.
- Biped Robust Standing: A complex 11-Degree-of-Freedom (DoF) simulated robot ("Johnnie") that must maintain its balance while being pelted by external forces.
- Object Grasping: A 7-DoF robot arm that must learn to reach and grasp an object from a table.
- Ball Catching: A 3-DoF arm with a 4-finger hand that must catch a falling ball, requiring precise, smooth timing.
Key Results and Discussion
Experimental results indicate that parameter-based methods offer several distinct advantages:
- Faster Convergence: On the Pole Balancing task, NES and PGPE significantly outperformed standard REINFORCE and eNAC (Episodic Natural Actor-Critic). NES converged the fastest overall.
- Robustness in High Dimensions: In the complex Biped Robust Standing task (which has 1,111 parameters), PGPE found a stable posture in under 1,000 episodes. Standard REINFORCE took twice as long to reach the same performance level.
- SDE as a "Magic Bullet": In the highly sensitive Ball Catching task, standard REINFORCE and eNAC frequently failed to learn a good policy. However, simply swapping their standard exploration for the SDE converter allowed them to reliably find superior catching behaviors with drastically lower variance.
By fixing the policy weights for an entire episode, parameter exploration ensures that actions come from a single, consistent function. This eliminates the "jitter" of action-exploration, making it vastly superior for physical robotics tasks like grasping or catching.
Conclusion
Parameter-based exploration bridges the gap between Reinforcement Learning and black-box optimization. By treating an entire episode as a single fitness evaluation, RL can borrow powerful techniques from the optimization field. While parameter-exploring methods can occasionally start slower in extremely high-dimensional spaces, their ability to provide smooth trajectories and simplified credit assignment makes them a powerful tool for modern robotics. Future work will explore integrating these methods into continuous value-based RL frameworks.
Summary updated