ProLoNets: Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning

Andrew Silva, Matthew C. GombolayarXiv 2019 · arXiv

Introduction: Overcoming the "Data Hunger" of Reinforcement Learning

Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet it remains plagued by a significant "data hunger" problem. Most RL agents begin as tabula rasa (blank slates), requiring millions of random trial-and-error interactions to learn even the most basic environmental concepts. While Imitation Learning (IL) offers a potential solution by mimicking human behavior, it necessitates massive, often unavailable datasets of expert demonstrations.

Furthermore, humans often find it easier to provide high-level symbolic logic (e.g., "If the enemy is close, retreat") rather than thousands of labeled state-action pairs. The challenge lies in bridging the gap between this intuitive, symbolic human logic and the differentiable weights of a neural network. This paper introduces ProLoNets (Propositional Logic Nets), a framework designed to "warm start" RL agents by directly encoding human expert knowledge into their architecture.

The ProLoNets Framework

ProLoNets transform human-readable decision trees into a fully differentiable neural network architecture. This process allows agents to skip the initial phase of random exploration and begin training from a position of informed competence.

The ProLoNets workflow: from human-specified rules to a differentiable policy improved via reinforcement learning.

Neural Encoding and Differentiable Logic

The core of ProLoNets is the translation of propositional rules into a neural structure. As shown in the architecture mapping below, decision nodes in a human-specified tree are converted into linear layers with sigmoid activations, while the leaves represent action weights.

Encoding a human expert's decision tree into a differentiable ProLoNet architecture.

Unlike traditional "hard" decision trees, ProLoNets are differentiable. This allows the agent to utilize standard RL algorithms, such as Proximal Policy Optimization (PPO), to fine-tune the initial human rules. Consequently, the human knowledge serves as a performance "floor" rather than a "ceiling," enabling the agent to eventually discover strategies that outperform the original instructions.

Dynamic Growth Mechanism

To ensure the agent is not limited by the complexity of the initial human logic, ProLoNets incorporate a dynamic growth mechanism. If the agent determines that its current rule set is insufficient for the environment's complexity, it can automatically deepen the network. By adding new nodes, the agent learns more nuanced behaviors that were not present in the original human specification.

Experimental Evaluation

The researchers evaluated ProLoNets across several challenging benchmarks:

  • StarCraft II (SC2): Including the FindAndDefeatZerglings minigame and the full game against the built-in AI.
  • Wildfire Tracking: A custom simulator where drones must track fires based on human-provided rules.
  • OpenAI Gym: Classic benchmarks like Cart Pole and Lunar Lander.

Key Results and Insights

  • StarCraft II Dominance: In the full game of StarCraft II, standard architectures (MLPs and LSTMs) failed to win a single match after 5,000 episodes. In contrast, ProLoNets initialized with basic strategic rules achieved a 100% win rate against the "Easy" AI and successfully learned to defeat the "Hard" AI.
  • Immediate Performance Boost: In the wildfire tracking domain, human-initialized agents provided an increase of over 80% in expected reward relative to random baselines before any training had even occurred.
  • The Value of Growth: In the Lunar Lander environment, the dynamic growth feature allowed the network to surpass initial human constraints, yielding twice the average reward of a static version of the same network.
  • Democratizing AI: A user study involving students with no specific training demonstrated that non-experts could use a simple interface to provide rules that significantly boosted agent performance ($p < 0.05$).

Conclusion and Future Directions

ProLoNets demonstrate that human domain knowledge can be effectively "injected" into neural networks to solve complex RL tasks that are otherwise intractable for tabula rasa agents. While the current model relies on semantically meaningful state spaces (e.g., distances and Boolean flags) and discrete action spaces, it paves the way for more accessible and efficient AI training. Future work will explore extending ProLoNets to raw pixel data via pre-trained feature extractors and adapting the framework for continuous control tasks.

Summary updated

Abstract

Deep reinforcement learning has been successful in a variety of tasks, such as game playing and robotic manipulation. However, attempting to learn \textit{tabula rasa} disregards the logical structure of many domains as well as the wealth of readily available knowledge from domain experts that could help "warm start" the learning process. We present a novel reinforcement learning technique that allows for intelligent initialization of a neural network weights and architecture. Our approach permits the encoding domain knowledge directly into a neural decision tree, and improves upon that knowledge with policy gradient updates. We empirically validate our approach on two OpenAI Gym tasks and two modified StarCraft 2 tasks, showing that our novel architecture outperforms multilayer-perceptron and recurrent architectures. Our knowledge-based framework finds superior policies compared to imitation learning-based and prior knowledge-based approaches. Importantly, we demonstrate that our approach can be used by untrained humans to initially provide >80% increase in expected reward relative to baselines prior to training (p < 0.001), which results in a >60% increase in expected reward after policy optimization (p = 0.011).
BibTeX
@misc{silva2019prolonets,
  author = {Andrew Silva and Matthew C. Gombolay},
  title = {{ProLoNets: Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning}},
  year = {2019},
  eprint = {1902.06007},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.06007},
}
Figure