Abstract
Propositional Logic Nets (PROLONETS) address the fundamental sample inefficiency of deep reinforcement learning (RL) by encoding human-provided "if-then" rules into a neural network's initial architecture and weights. This "warm start" approach allows agents to bypass the grueling random exploration phase, enabling them to tackle complex environments like StarCraft II where standard RL often fails. By combining human heuristics with dynamic network growth and gradient-based optimization, PROLONETS achieve superior performance and faster convergence across diverse domains.
Introduction
Most deep reinforcement learning (RL) agents start tabula rasa—as a blank slate. While this allows for unbiased learning, it is incredibly inefficient. In complex domains with sparse rewards, an agent might spend millions of steps wandering randomly before stumbling upon a useful signal.
Existing solutions to this problem have significant drawbacks. Imitation Learning (IL) requires massive datasets of expert demonstrations, which are expensive or impossible to collect for many tasks. Meanwhile, existing knowledge-based neural networks often use human knowledge to define a network's structure but still require the agent to learn the actual logic from scratch. The authors of this paper address the challenge of leveraging high-level human expertise (e.g., "If the enemy is close, attack; otherwise, retreat") to give an RL agent a functional starting policy without needing a pre-existing dataset of millions of labeled state-action pairs.
The PROLONETS Framework
The authors introduce PROLONETS, a framework that translates human-specified decision trees into trainable neural networks. This allows domain experts to provide a "good enough" starting point that the agent can then refine through experience.
Intelligent Initialization
Human rules expressed in propositional logic are mapped directly to the network architecture. Decision nodes in the human's tree become linear layers with sigmoid activations, while leaves are mapped to action weights.
A key parameter in this mapping is $\alpha$, which controls the "crispness" of the logic. A high $\alpha$ makes the node act like a strict Boolean gate, mirroring the original human logic. A lower $\alpha$ allows for "fuzzier" decision-making, which facilitates better gradient flow during the training phase, allowing the network to adapt more easily.
Dynamic Growth
To prevent the agent from being limited by potentially incomplete human logic, PROLONETS can grow dynamically. The system maintains a "deeper" version of the network in the background. If this deeper version shows lower entropy—meaning it has become more certain and effective than the current shallow version—the network "deepens" itself at that specific node. This allows the AI to expand upon the human's initial scaffolding as it discovers more complex strategies.
Optimization
Once initialized, the agent is trained using Proximal Policy Optimization (PPO). This stage allows the agent to fine-tune the human's initial heuristics into an optimal strategy, effectively using RL to polish the "rough" human input.
Experimental Evaluation
The researchers evaluated PROLONETS across several benchmarks:
- OpenAI Gym: Classic control tasks like "Cart Pole" and "Lunar Lander."
- StarCraft II (SC2LE): A "FindAndDefeatZerglings" mini-game and a full game against the built-in AI.
- Wildfire Tracking: A custom-built domain where two drones must coordinate to track fire centroids in a 500x500 grid.
Key Results and Insights
- StarCraft II Dominance: In the full game of StarCraft II, PROLONETS was the only architecture capable of winning. Baselines like standard MLPs, LSTMs, and even other knowledge-based architectures (DJINN) failed to win a single game. PROLONETS achieved a 100% win rate against "easy" AI and doubled its win rate against "hard" AI after training.
- Efficiency Gains: In the StarCraft mini-game, PROLONETS obtained over 100% more average reward than baseline RL approaches that started from scratch.
- Human-in-the-Loop Success: In the wildfire domain, a user study with non-experts showed that policies specified via a simple UI resulted in an 80% increase in expected reward relative to baselines before any training occurred. After training, these human-started agents maintained a 60% lead over agents that started from scratch.
- Surpassing the Teacher: The results suggest that human logic serves as excellent scaffolding. The AI uses human heuristics to get into the "right neighborhood" of the solution and then uses its own computational power to explore further and surpass the human's initial performance.
Conclusion and Future Work
PROLONETS demonstrate that encoding domain knowledge can democratize AI, allowing experts like pilots or doctors to initialize systems without deep computer science expertise. However, some limitations remain. The current implementation relies on semantically meaningful state spaces (e.g., coordinates and distances) rather than raw pixels. Future work will focus on extending this approach to continuous action spaces, such as joint torques in robotics, and further improving the architecture's robustness to sub-optimal or incorrect human initializations.
Summary updated