Adaptive Variance for Changing Sparse-Reward Environments

Xingyu Lin, Pengsheng Guo, Carlos Florensa, David Held

Abstract

Robots that are trained to perform a task in a fixed environment often fail when facing unexpected changes to the environment due to a lack of exploration. We propose a principled way to adapt the policy for better exploration in changing sparse-reward environments. Unlike previous works which explicitly model environmental changes, we analyze the relationship between the value function and the optimal exploration for a Gaussian-parameterized policy and show that our theory leads to an effective strategy for adjusting the variance of the policy, enabling fast adapt to changes in a variety of sparse-reward environments.

1. Introduction

Reinforcement Learning (RL) has achieved impressive feats in robotics, but usually under the assumption that the environment is static. In the real world, environments are dynamic: a table's friction may change, a robot’s joints wear down, or target objects are moved.

A visualization of environmental change in the robot simulation, where the task surface properties or geometry shift mid-training.

When a robot is trained in a "sparse-reward" environment (receiving binary feedback of 1 for success and 0 for failure), it faces a catch-22. Once it masters a task, its exploration variance typically shrinks to zero to maximize performance. If the environment then shifts, the robot fails, receives no reward, and because it has stopped exploring, it never discovers the new solution. This creates a vicious cycle where poor exploration leads to poor updates, preventing adaptation to distributional shifts.

2. Methodology: Value Dependent Exploration (VD)

The core contribution of this work is Value Dependent Exploration (VD). The method is built on the principle that an agent should use its current estimate of success (the value function) to determine its exploration variance.

2.1 Theoretical Foundation

Starting with a continuous bandit model, we derive that the optimal variance $\sigma^*$ is inversely related to the value function $V$. Specifically, $\sigma^* \approx \frac{w}{\sqrt{2\pi e V}}$. This generalizes to complex Markov Decision Processes (MDPs), where we prove that the mapping from value to variance should be monotonically decreasing. If the expected reward (Value) drops, exploration (Variance) must increase.

2.2 Implementation

To implement this, we utilize a VD-Sigmoid function to map the state-dependent value $V(s)$ to an action variance $\sigma(s)$. This function is trained alongside the policy using standard policy gradient methods like TRPO. During rollouts, the robot samples actions from a Gaussian distribution where the mean is provided by the policy and the variance is calculated dynamically based on the current state's value.

3. Experimental Setup

We evaluated VD on several challenging continuous-action environments simulated in MuJoCo:

  • Double Inverted Pendulum (DIP): Balancing a pole with a sudden shift in the center of mass.
  • Thrower: A 7-DOF robot arm throwing a ball where the base position is moved mid-training.
  • Ball in Cup (BiC): Swinging a ball into a cup with changing orientation and height.
  • Fetch Slide: Sliding a block to a target with altered table friction.

4. Results and Discussion

In all MuJoCo tasks, the VD-Sigmoid method demonstrated rapid recovery following environmental changes. As shown in the training curves, when the environment shifts at iteration 2000, the VD method's entropy (variance) jumps immediately, allowing it to re-explore and regain high success rates.

Training curves showing average reward, action mean, and action variance. At iteration 2000, the environment changes; VD automatically increases variance to re-explore and recover, whereas VPG remains stuck.

4.1 Comparison to Baselines

  • Vanilla Policy Gradient (VPG): Fails to adapt because its variance remains near zero after the initial training phase.
  • Relearn: Even a baseline that resets the model from scratch upon detecting a change performs worse than VD. VD retains useful prior knowledge in the policy mean while only increasing variance to find the new optimal parameters.
  • Curiosity-based (VIME): While effective at exploration, it can be unstable during sudden dynamics shifts compared to the principled VD approach.

5. Conclusion

Value Dependent Exploration provides a mathematically grounded way to break the vicious cycle of poor exploration in changing environments. By treating the value function as a proxy for environmental shift, robots can automatically ramp up exploration exactly when needed. Future work will focus on extending this derivation to dense-reward settings and scenarios requiring entirely new behavioral strategies rather than simple parameter translations.

Summary updated

Abstract

Robots that are trained to perform a task in a fixed environment often fail when facing unexpected changes to the environment due to a lack of exploration. We propose a principled way to adapt the policy for better exploration in changing sparse-reward environments. Unlike previous works which explicitly model environmental changes, we analyze the relationship between the value function and the optimal exploration for a Gaussian-parameterized policy and show that our theory leads to an effective strategy for adjusting the variance of the policy, enabling fast adapt to changes in a variety of sparse-reward environments.
BibTeX
@inproceedings{lin2019adaptive,
  author = {Xingyu Lin and Pengsheng Guo and Carlos Florensa and David Held},
  title = {{Adaptive Variance for Changing Sparse-Reward Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=_52TYST-5o6},
}
Figure