Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling

Yuping Luo, Huazhe Xu, Tengyu Ma

Abstract

Standard imitation learning techniques, such as Behavioral Cloning, frequently fail due to covariate shift: small errors accumulate, pushing the agent into states it never encountered during training. We propose Value Iteration with Negative Sampling (VINS), an algorithm designed to learn "self-correctable" policies. By training a value function to be conservative—assigning lower values to states outside the expert distribution—we create a "value funnel" that naturally guides the agent back to successful trajectories. VINS achieves this using only static demonstrations, avoiding the need for costly environment interactions during training.

The Challenge: Covariate Shift and Optimistic Extrapolation

The primary limitation of Behavioral Cloning (BC) is its inability to recover from errors. Trained only on "perfect" expert paths, the agent lacks guidance when it drifts into slightly different states. This leads to cascading errors and eventual failure.

Furthermore, learning a value function from static data often results in optimistic extrapolation, where the model erroneously assigns high values to unseen, potentially dangerous states. If an agent follows these "fake" high-value signals, it will move further away from the expert path.

Methodology

The authors introduce a two-pronged strategy to induce self-correction:

1. Conservative Extrapolation

VINS defines a value function that matches expert values on demonstration states but is explicitly regularized to decrease as the agent moves away from those states. This creates a geometric "funnel" effect: the gradient of the value function always points back toward the expert states.

Conceptual illustration of the self-correcting value function. The value is forced to be lower outside the expert distribution U, creating a gradient that guides the agent back to the demonstration path.

2. Negative Sampling

To enforce this conservative behavior without active exploration, VINS employs Negative Sampling. Expert states are perturbed with noise to create synthetic "negative" states. The value function is then trained to minimize a loss that ensures these negative samples have lower values than the original expert states.

Visualization of the learned value function and induced policy. The red regions represent high value along the expert trajectory, while the blue regions represent lower values. The arrows show the policy naturally correcting back towards the high-value 'expert' states.

3. Induced Policy and Dynamics

Rather than training a separate policy network, VINS learns a dynamics model $M(s, a)$. At execution time, the agent evaluates several candidate actions (near the BC suggestion) and selects the one that leads to the predicted next state with the highest value.

Experimental Evaluation

The method was evaluated on three sparse-reward robotics benchmarks from OpenAI Gym and MuJoCo: Reach, Push, and Pick-And-Place.

Key Results

  • Offline Superiority: In purely offline settings, VINS significantly outperformed Behavioral Cloning. For instance, on the "Push" task with 100 demonstrations, VINS achieved a 44% success rate compared to 37.3% for BC.
  • RL Warm-Starting: When used to initialize Reinforcement Learning (VINS+RL), the method reached near-optimal performance much faster than existing state-of-the-art methods, effectively bypassing the inefficient initial exploration phase.
  • Ablation Study: Removing negative sampling (VINS w/o NS) led to a dramatic drop in performance (e.g., from 75.7% to 48.5% on Pick-And-Place), confirming that conservative extrapolation is critical for success.

Discussion and Future Work

While VINS shows promise, its reliance on Gaussian noise for negative sampling may struggle with high-dimensional inputs like raw pixels. Future research could explore performing negative sampling in a learned latent representation space. Additionally, extending the theoretical guarantees of self-correction to stochastic environments remains an important open challenge.

Summary updated

BibTeX
@inproceedings{luo2020learning,
  author = {Yuping Luo and Huazhe Xu and Tengyu Ma},
  title = {{Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure