Abstract
Standard imitation learning techniques, such as Behavioral Cloning, frequently fail due to covariate shift: small errors accumulate, pushing the agent into states it never encountered during training. We propose Value Iteration with Negative Sampling (VINS), an algorithm designed to learn "self-correctable" policies. By training a value function to be conservative—assigning lower values to states outside the expert distribution—we create a "value funnel" that naturally guides the agent back to successful trajectories. VINS achieves this using only static demonstrations, avoiding the need for costly environment interactions during training.
The Challenge: Covariate Shift and Optimistic Extrapolation
The primary limitation of Behavioral Cloning (BC) is its inability to recover from errors. Trained only on "perfect" expert paths, the agent lacks guidance when it drifts into slightly different states. This leads to cascading errors and eventual failure.
Furthermore, learning a value function from static data often results in optimistic extrapolation, where the model erroneously assigns high values to unseen, potentially dangerous states. If an agent follows these "fake" high-value signals, it will move further away from the expert path.
Methodology
The authors introduce a two-pronged strategy to induce self-correction:
1. Conservative Extrapolation
VINS defines a value function that matches expert values on demonstration states but is explicitly regularized to decrease as the agent moves away from those states. This creates a geometric "funnel" effect: the gradient of the value function always points back toward the expert states.
2. Negative Sampling
To enforce this conservative behavior without active exploration, VINS employs Negative Sampling. Expert states are perturbed with noise to create synthetic "negative" states. The value function is then trained to minimize a loss that ensures these negative samples have lower values than the original expert states.
3. Induced Policy and Dynamics
Rather than training a separate policy network, VINS learns a dynamics model $M(s, a)$. At execution time, the agent evaluates several candidate actions (near the BC suggestion) and selects the one that leads to the predicted next state with the highest value.
Experimental Evaluation
The method was evaluated on three sparse-reward robotics benchmarks from OpenAI Gym and MuJoCo: Reach, Push, and Pick-And-Place.
Key Results
- Offline Superiority: In purely offline settings, VINS significantly outperformed Behavioral Cloning. For instance, on the "Push" task with 100 demonstrations, VINS achieved a 44% success rate compared to 37.3% for BC.
- RL Warm-Starting: When used to initialize Reinforcement Learning (VINS+RL), the method reached near-optimal performance much faster than existing state-of-the-art methods, effectively bypassing the inefficient initial exploration phase.
- Ablation Study: Removing negative sampling (VINS w/o NS) led to a dramatic drop in performance (e.g., from 75.7% to 48.5% on Pick-And-Place), confirming that conservative extrapolation is critical for success.
Discussion and Future Work
While VINS shows promise, its reliance on Gaussian noise for negative sampling may struggle with high-dimensional inputs like raw pixels. Future research could explore performing negative sampling in a learned latent representation space. Additionally, extending the theoretical guarantees of self-correction to stochastic environments remains an important open challenge.
Summary updated