Abstract
Deep Reinforcement Learning (DRL) has achieved human-level performance in complex tasks, but the underlying decision-making logic is often obscured within deep neural networks. This paper introduces Linear Model U-Trees (LMUTs), a mimic learning framework that approximates DRL policies with an interpretable tree structure. By utilizing linear models at the leaf nodes, LMUTs provide a transparent mechanism for rule extraction and feature influence analysis without compromising significant performance.
Introduction
While DRL excels in games and robotics, its "black box" nature poses a significant barrier to deployment in safety-critical fields such as healthcare or autonomous driving. The "knowledge" acquired by these agents is typically buried in millions of neural network weights. Existing mimic learning techniques, which train a simpler model to replicate a complex one, have primarily focused on static supervised learning tasks. Reinforcement learning, however, is a dynamic process involving interaction with an environment. This research addresses this gap by developing a mimicry framework specifically for the Q-functions of DRL agents.
The LMUT Framework
The proposed Linear Model U-Tree (LMUT) builds upon the classic U-Tree algorithm with several key innovations designed for the RL context.
Linear Leaf Nodes
Unlike standard regression trees that predict a constant value at each leaf, LMUTs employ a linear regression model at every leaf node. This architectural choice significantly enhances the tree's expressivity, allowing it to approximate continuous functions with fewer branches. This sparsity is crucial for maintaining human interpretability.
Active Play and Online Learning
The authors introduce an "Active Play" setting where the mimic learner observes a well-trained DRL agent in real-time. Instead of learning from a static dataset, the LMUT updates its parameters as the agent interacts with the environment.
The learning process utilizes a novel online algorithm. It employs Stochastic Gradient Descent (SGD) to update the linear models at the leaves and a variance-based splitting criterion to grow the tree in regions where prediction uncertainty is highest.
Experimental Evaluation
The LMUT was evaluated on three OpenAI Gym benchmarks:
- Mountain Car: A physics-based task requiring momentum management.
- Cart Pole: A classic balancing challenge.
- Flappy Bird: A high-dimensional task using 80x80 binary pixel images.
Key Insights and Results
The research yielded several critical findings regarding the trade-offs between model fidelity and actual play performance.
- Performance vs. Fidelity: High fidelity (closeness to the neural network's output) does not always translate to high play performance. LMUT succeeds by balancing general state-space coverage with high accuracy in critical decision states.
- Superior Play Performance: In the Active Play setting, LMUT consistently outperformed baselines like CART and M5 trees. Notably, in Flappy Bird, LMUT achieved a score of 103.32, closely approaching the original DQN's 123.42, whereas other mimic models failed to perform effectively.
- Interpretability and Rule Extraction: The model successfully identified "Pole Angle" and "Velocity" as the most critical features for Cart Pole and Mountain Car, respectively. Furthermore, the researchers demonstrated the ability to extract human-readable "if-then" rules, explaining complex behaviors like building kinetic energy in Mountain Car.
Conclusion and Future Work
LMUTs represent a significant step toward transparent DRL. By combining the hierarchical structure of trees with the local precision of linear models, they offer a viable path for auditing AI decisions. Future research will focus on implementing sophisticated tree pruning to further enhance simplicity, exploring proactive sampling strategies to better cover the state space, and scaling the approach to even higher-dimensional continuous action spaces.
Summary updated