Encoding Human Domain Knowledge to Warm Start Reinforcement Learning

Andrew Silva, Matthew C. Gombolay

Abstract

Propositional Logic Nets (PROLONETS) address the fundamental sample inefficiency of deep reinforcement learning (RL) by encoding human-provided "if-then" rules into a neural network's initial architecture and weights. This "warm start" approach allows agents to bypass the grueling random exploration phase, enabling them to tackle complex environments like StarCraft II where standard RL often fails. By combining human heuristics with dynamic network growth and gradient-based optimization, PROLONETS achieve superior performance and faster convergence across diverse domains.

Introduction

Most deep reinforcement learning (RL) agents start tabula rasa—as a blank slate. While this allows for unbiased learning, it is incredibly inefficient. In complex domains with sparse rewards, an agent might spend millions of steps wandering randomly before stumbling upon a useful signal.

Existing solutions to this problem have significant drawbacks. Imitation Learning (IL) requires massive datasets of expert demonstrations, which are expensive or impossible to collect for many tasks. Meanwhile, existing knowledge-based neural networks often use human knowledge to define a network's structure but still require the agent to learn the actual logic from scratch. The authors of this paper address the challenge of leveraging high-level human expertise (e.g., "If the enemy is close, attack; otherwise, retreat") to give an RL agent a functional starting policy without needing a pre-existing dataset of millions of labeled state-action pairs.

The PROLONETS Framework

The authors introduce PROLONETS, a framework that translates human-specified decision trees into trainable neural networks. This allows domain experts to provide a "good enough" starting point that the agent can then refine through experience.

The PROLONETS workflow, from human-specified rules to RL-based policy improvement.

Intelligent Initialization

Human rules expressed in propositional logic are mapped directly to the network architecture. Decision nodes in the human's tree become linear layers with sigmoid activations, while leaves are mapped to action weights.

Mapping a human expert's decision tree policy into the PROLONET neural network architecture.

A key parameter in this mapping is $\alpha$, which controls the "crispness" of the logic. A high $\alpha$ makes the node act like a strict Boolean gate, mirroring the original human logic. A lower $\alpha$ allows for "fuzzier" decision-making, which facilitates better gradient flow during the training phase, allowing the network to adapt more easily.

Dynamic Growth

To prevent the agent from being limited by potentially incomplete human logic, PROLONETS can grow dynamically. The system maintains a "deeper" version of the network in the background. If this deeper version shows lower entropy—meaning it has become more certain and effective than the current shallow version—the network "deepens" itself at that specific node. This allows the AI to expand upon the human's initial scaffolding as it discovers more complex strategies.

Optimization

Once initialized, the agent is trained using Proximal Policy Optimization (PPO). This stage allows the agent to fine-tune the human's initial heuristics into an optimal strategy, effectively using RL to polish the "rough" human input.

Experimental Evaluation

The researchers evaluated PROLONETS across several benchmarks:

  • OpenAI Gym: Classic control tasks like "Cart Pole" and "Lunar Lander."
  • StarCraft II (SC2LE): A "FindAndDefeatZerglings" mini-game and a full game against the built-in AI.
  • Wildfire Tracking: A custom-built domain where two drones must coordinate to track fire centroids in a 500x500 grid.

Key Results and Insights

  • StarCraft II Dominance: In the full game of StarCraft II, PROLONETS was the only architecture capable of winning. Baselines like standard MLPs, LSTMs, and even other knowledge-based architectures (DJINN) failed to win a single game. PROLONETS achieved a 100% win rate against "easy" AI and doubled its win rate against "hard" AI after training.
  • Efficiency Gains: In the StarCraft mini-game, PROLONETS obtained over 100% more average reward than baseline RL approaches that started from scratch.
  • Human-in-the-Loop Success: In the wildfire domain, a user study with non-experts showed that policies specified via a simple UI resulted in an 80% increase in expected reward relative to baselines before any training occurred. After training, these human-started agents maintained a 60% lead over agents that started from scratch.
  • Surpassing the Teacher: The results suggest that human logic serves as excellent scaffolding. The AI uses human heuristics to get into the "right neighborhood" of the solution and then uses its own computational power to explore further and surpass the human's initial performance.

Conclusion and Future Work

PROLONETS demonstrate that encoding domain knowledge can democratize AI, allowing experts like pilots or doctors to initialize systems without deep computer science expertise. However, some limitations remain. The current implementation relies on semantically meaningful state spaces (e.g., coordinates and distances) rather than raw pixels. Future work will focus on extending this approach to continuous action spaces, such as joint torques in robotics, and further improving the architecture's robustness to sub-optimal or incorrect human initializations.

Summary updated

Abstract

Deep reinforcement learning has been successful in a variety of tasks, such as game playing and robotic manipulation. However, attempting to learn tabula rasa disregards the logical structure of many domains as well as the wealth of readily available knowledge from domain experts that could help "warm start" the learning process. We present a novel reinforcement learning technique that allows for intelligent initialization of a neural network weights and architecture. Our approach permits the encoding domain knowledge directly into a neural decision tree, and improves upon that knowledge with policy gradient updates. We empirically validate our approach on two OpenAI Gym tasks and two modified StarCraft 2 tasks, showing that our novel architecture outperforms multilayer-perceptron and recurrent architectures. Our knowledge-based framework finds superior policies compared to imitation learning-based and prior knowledge-based approaches. Importantly, we demonstrate that our approach can be used by untrained humans to initially provide >80% increase in expected reward relative to baselines prior to training (p < 0.001), which results in a >60% increase in expected reward after policy optimization (p = 0.011).
BibTeX
@inproceedings{silva2021encoding,
  author = {Andrew Silva and Matthew C. Gombolay},
  title = {{Encoding Human Domain Knowledge to Warm Start Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i6.16638},
}
Figure