V-MIN: Efficient Reinforcement Learning through Demonstrations and Relaxed Reward Demands

David Martínez Martínez, Guillem Alenyà, Carme TorrasAAAI 2015 · DOI

Abstract

Reinforcement learning (RL) in robotics is frequently constrained by the high cost of exploration. This paper introduces V-MIN, an algorithm that accelerates learning by incorporating a user-defined performance threshold, $V_{min}$. When an agent determines it cannot meet this threshold autonomously, it proactively requests a human demonstration. This approach balances self-discovery with expert guidance, enabling faster convergence to functional policies in complex, relational domains.

Introduction

Reinforcement Learning is a powerful tool for robotics, but it faces a significant "speed limit" due to exploration requirements. For a robot to learn a high-level task, such as clearing a table, it typically must perform thousands of random actions to discover a reward. In physical environments, this process is slow, causes mechanical wear, and often leads to sub-optimal or risky strategies. Existing methods like R-MAX encourage exploration through optimism but can spend excessive time in irrelevant state spaces. V-MIN addresses this by allowing a robot to identify when it is stuck and request targeted human assistance.

Methodology

V-MIN builds upon the R-MAX framework (optimism under uncertainty) and REX-D (relational logic for generalization). The core innovation is the $V_{min}$ parameter, which defines a "good enough" reward threshold.

  1. Relational Learning: The robot utilizes Noisy Deictic Rules (NDR) to learn general rules for object classes (e.g., "Mugs") rather than specific instances.
  2. Quality Threshold ($V_{min}$): If the robot's internal planner calculates that it cannot reach a state with a value $\ge V_{min}$ using current knowledge, it triggers a TeacherRequest().
  3. Active Learning: Upon triggering a request, the human teacher provides a demonstration of the next best move, which is then integrated into the robot's relational model to fill knowledge gaps.
Evolution of learned rules and value estimates across iterations.

Experimental Evaluation

The authors evaluated V-MIN across three simulated domains:

  • Triangle Tireworld: A planning problem with a risky short path and a safe long path.
  • Exploding Blocksworld: A stacking task where moves can permanently destroy objects.
  • Table Clearing: A robotics domain requiring the stacking and transport of varied items like plates, cups, and forks.

Key Results and Insights

V-MIN demonstrates that relaxing the demand for an absolute optimal policy in favor of a functional one significantly accelerates training. In the Exploding Blocksworld, V-MIN reached an optimal policy with an average of only 14 exploration actions, compared to approximately 250 for the REX algorithm.

Performance comparison of V-MIN against REX and REX-D in terms of actions and success rate.

Furthermore, V-MIN enhances safety. In the Tireworld experiment, V-MIN was the only algorithm to consistently choose the safe path. Because the risky path's high failure probability resulted in an expected value below $V_{min}$, the robot was prompted to seek demonstrations that eventually revealed the safer route. The use of relational logic also allowed for immediate knowledge reuse across similar objects, further reducing sample complexity.

Conclusion

V-MIN effectively reduces the exploration burden in RL by allowing agents to signal for help when performance expectations are not met. While current limitations include the need for a pre-defined reward function and full observability, future work may involve learning rewards directly from demonstrations and adapting the model for partially observable environments.

Summary updated

BibTeX
@inproceedings{martinez2015v,
  author = {David Martínez Martínez and Guillem Alenyà and Carme Torras},
  title = {{V-MIN: Efficient Reinforcement Learning through Demonstrations and Relaxed Reward Demands}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9596},
}
Figure