Abstract
In explainable artificial intelligence, there is increasing interest in understanding the behaviour of autonomous agents to build trust and validate performance. Modern agent architectures, such as those trained by deep reinforcement learning, are currently so lacking in interpretable structure as to effectively be black boxes, but insights may still be gained from an external, behaviourist perspective. Inspired by conceptual spaces theory, we suggest that a versatile first step towards general understanding is to discretise the state space into convex regions, jointly capturing similarities over the agent's action, value function and temporal dynamics within a dataset of observations. We create such a representation using a novel variant of the CART decision tree algorithm, and demonstrate how it facilitates practical understanding of black box agents through prediction, visualisation and rule-based explanation.
Introduction
Modern autonomous agents, particularly those trained via Deep Reinforcement Learning (DRL), operate as "black boxes" with internal logic hidden within millions of neural network parameters. While existing Explainable AI (XAI) tools often focus on policy imitation—predicting an agent's next action—this approach provides limited insight into why an agent behaves a certain way. To fully validate safety-critical systems, we must understand how an agent perceives its own performance (its Value Function) and how it expects the environment to change (Environment Dynamics). TripleTree addresses this by proposing a unified model that captures policy, value, and dynamics simultaneously.
The TripleTree Framework
TripleTree is an extension of the Classification and Regression Trees (CART) algorithm designed to partition continuous state spaces into "conceptual regions." Unlike standard trees that minimize impurity for a single label, TripleTree employs a hybrid impurity measure considering three distinct attributes:
- Action ($A$): The agent's chosen behavior.
- Value ($V$): The expected future reward from a given state.
- Derivatives ($D$): The change in state features over a single timestep.
A weight vector $\theta$ allows for tuning the model's focus. For example, a balanced weighting creates a versatile model capable of multi-modal explanations, including factual, counterfactual, and temporal reasoning. By calculating transition probabilities between the resulting leaves, the system converts continuous dynamics into a discrete Probabilistic Finite State Machine (FSM).
Empirical Evaluation and Key Results
The framework was evaluated on a custom 2D Road MDP and the LunarLanderContinuous-v2 benchmark. A high-performing Soft Actor-Critic (SAC) agent served as the black-box target.
Synergy of Objectives
A key finding is the synergy between predicting dynamics and actions; partitioning based on state derivatives proved highly effective for policy prediction. However, modeling the value function proved more complex, requiring a higher density of leaves to achieve accuracy comparable to policy or dynamics modeling.
Discovering Hidden Behaviors
In the LunarLander environment, TripleTree exposed asymmetries in the SAC agent's policy that were not apparent through standard testing. Despite a symmetric training environment, the agent exhibited less efficient, "jittery" behavior when approaching from the right compared to the left. Such insights demonstrate TripleTree's utility in identifying subtle behavioral bugs.
Discussion and Limitations
While TripleTree provides a structured view of agent logic, it currently relies on semantically meaningful input features (e.g., velocity) and cannot directly process raw pixel data without pre-processing. Additionally, while the model scales well to 8D spaces, extremely high-dimensional environments may lead to fragmented, less interpretable trees. Future work will focus on refining trajectory analysis and applying the model to multi-agent systems.
Summary updated