Adaptive Variance for Changing Sparse-Reward Environments

Xingyu Lin, Pengsheng Guo, Carlos Florensa, David Held

TL;DR

This paper introduces a principled method for robots to automatically ramp up exploration when their environment changes. By mathematically linking a policy's "value function" to its optimal exploration variance, the authors enable agents to quickly recover from performance drops in difficult, sparse-reward tasks.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has achieved impressive feats in robotics, but usually under the assumption that the environment is static. In the real world, things change: a table gets slippery, a robot’s joints wear down, or an object is moved.

When a robot is trained in a "sparse-reward" environment (where it only gets a "1" for success and "0" otherwise), it faces a catch-22. Once it masters a task, its exploration variance typically shrinks to zero to maximize performance. If the environment then shifts, the robot fails, receives no reward, and because it has stopped exploring, it never "stumbles" upon the new solution. The authors call this a vicious cycle where poor exploration leads to poor updates, preventing the robot from adapting to "distributional shifts."

Methods

The authors propose Value Dependent Exploration (VD). The core idea is simple but mathematically grounded: the agent should use its current estimate of success (the value function) to decide how much to explore.

  1. Theoretical Derivation: Starting with a "continuous bandit" (a simplified one-step task), the authors prove that the optimal variance $\sigma^*$ is inversely related to the value function $V$. Specifically, they derive a relationship where $\sigma^* \approx \frac{w}{\sqrt{2\pi e V}}$.
  2. Monotonic Mapping: They generalize this to more complex Markov Decision Processes (MDPs), proving that the mapping from value to variance should be monotonically decreasing. If your expected reward (Value) drops, your exploration (Variance) must go up.
  3. VD-Sigmoid: To implement this, they use a sigmoid-based function to map the state-dependent value $V(s)$ to an action variance $\sigma(s)$. This function is trained alongside the policy using standard policy gradient methods (like TRPO).
  4. The Workflow: During rollouts, the robot samples actions from a Gaussian distribution where the mean comes from the policy and the variance is calculated on-the-fly based on the current state's value.

Datasets (Environments)

The researchers evaluated their method on several challenging continuous-action environments simulated in MuJoCo:

  • Continuous Bandit: A 1D toy problem to verify the theory.
  • Double Inverted Pendulum (DIP): Balancing a pole where the center of mass is suddenly shifted.
  • Thrower: A 7-DOF robot arm throwing a ball into a basket; the robot's base position is moved mid-training.
  • Ball in Cup (BiC): Swinging a ball into a cup; the cup's orientation and height are changed.
  • Fetch Slide: A robot sliding a block to a target; the friction coefficient of the table is altered.

Key Insights and Ideas

  • Value as a Measure of Shift: In binary sparse-reward settings, the value function (ranging from 0 to 1) is a perfect proxy for environmental shift. A value of 1 means no shift; a value of 0 means a drastic shift requiring maximum exploration.
  • Breaking the Vicious Cycle: Unlike "count-based" exploration (which stops exploring once a state is "visited" many times), VD keeps exploring as long as performance is poor, even in familiar states.
  • The Power of Monotonicity: Constraining the mapping function to be monotonically decreasing (via the VD-Sigmoid) is crucial. Unconstrained neural networks (VD-NN) struggle to learn the complex relationship between performance and variance in high-dimensional spaces.

Key Results

  • Rapid Recovery: In all MuJoCo tasks, when the environment changed, the VD-Sigmoid method's entropy (exploration) jumped immediately. This allowed it to recover and reach high success rates much faster than baselines.
  • Outperforming Baselines:
    • Global/Fixed Variance: These methods either explore too little to find the new solution or too much to ever perform precisely.
  • VIME (Curiosity-based): While VIME explores well, it can be unstable when dynamics change suddenly.
  • Relearn: Even a "perfect" detector that resets the model from scratch when a change occurs performed worse than VD, because VD retains useful prior knowledge (like the "mean" action) while only increasing variance.

Limitations and Future Work

  • Sparse Rewards Only: The current mathematical derivation relies on binary sparse rewards (0 or 1). Extending this to "dense" rewards (where the robot gets constant feedback) is an open challenge.
  • Translation Assumption: The theory assumes that environmental changes primarily result in a "translation" of the optimal action space (e.g., you need to push a bit harder because of friction, but the motion is similar). It may struggle with shifts that require entirely different strategies.
  • Future Direction: The authors suggest exploring how these strategies could lead to more robust robot learning in real-world deployments where environmental variables are constantly in flux.

Summary updated

Abstract

Robots that are trained to perform a task in a fixed environment often fail when facing unexpected changes to the environment due to a lack of exploration. We propose a principled way to adapt the policy for better exploration in changing sparse-reward environments. Unlike previous works which explicitly model environmental changes, we analyze the relationship between the value function and the optimal exploration for a Gaussian-parameterized policy and show that our theory leads to an effective strategy for adjusting the variance of the policy, enabling fast adapt to changes in a variety of sparse-reward environments.
BibTeX
@inproceedings{lin2019adaptive,
  author = {Xingyu Lin and Pengsheng Guo and Carlos Florensa and David Held},
  title = {{Adaptive Variance for Changing Sparse-Reward Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=_52TYST-5o6},
}

Loading related content...

Figure