Interpretable Policies for Sequential Decision Making

539 concepts470 papers3 proposals

Deep Reinforcement Learning (DRL) has achieved remarkable success in complex sequential decision-making tasks, from mastering games to optimizing industrial workflows. However, these models typically operate as "black boxes," hiding their decision-making logic within millions of uninterpretable neural network weights. In safety-critical or highly regulated domains—such as autonomous driving, healthcare, and power grid management—this opacity is a major liability. As George Vouros et al. categorize in their overview of Explainable Deep Reinforcement Learning: State of the Art and Challenges, achieving explainable agency requires moving beyond raw performance toward methods that expose global policy logic, explain local outcomes, and allow internal model inspection.

Distilling Opaque Policies into Symbolic Structures

Why do neural networks lack transparency? They draw smooth, high-dimensional decision boundaries that humans cannot easily parse. One prominent strategy in Explainable Reinforcement Learning (XRL) is to replace or approximate these complex boundaries with compact, symbolic representations that map states to actions via discrete, human-readable rules.

For instance, Louis Wehenkel et al. demonstrate in Policy Search in a Space of Simple Closed-form Formulas: Towards Interpretability of Reinforcement Learning that policy search can be restricted entirely to a space of simple, closed-form mathematical formulas. By framing this search as a multi-armed bandit problem, they discover compact equations that match or exceed the performance of deep planning models. Similarly, Hengzhe Zhang et al. use Evolutionary Feature Synthesis in Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis to distill opaque neural network controllers into transparent symbolic expressions.

Another approach mimics the policy using tree structures, which naturally map states to actions via discrete, hierarchical rules. Guiliang Liu et al. introduce Linear Model U-Trees (LMUTs) in Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees to approximate deep reinforcement learning policies, placing easily auditable linear models at the leaf nodes. To scale this to continuous control domains like autonomous driving, Rohan Paleja et al. propose Interpretable Continuous Control Trees (ICCTs) in Learning Interpretable, High-Performing Policies for Autonomous Driving, which achieve performance parity with deep neural networks while using orders of magnitude fewer parameters. Going a step further, Wenjie Qiu et al. eliminate the need for an initial neural network "teacher" entirely in Programmatic Reinforcement Learning without Oracles, synthesizing human-readable programmatic policies directly from reward signals via differentiable architecture search.

Causal Reasoning and Contrastive Explanations

While symbolic formulas and trees show what rule was triggered, they do not inherently explain the underlying cause and effect. When an autonomous agent takes an unexpected action, human operators typically ask contrastive questions: "Why did you choose action A instead of action B?" Answering this requires a causal model of the environment.

To address this, Prashan Madumal et al. integrate Structural Causal Models (SCMs) into the reinforcement learning process in Explainable Reinforcement Learning through a Causal Lens. By learning an "action influence model" during training, the agent maps how its actions causally affect environmental variables, enabling it to generate human-like contrastive explanations. This causal induction can also be driven by external models; Alexander Dockhorn et al. show in From Gameplay Traces to Game Mechanics: Causal Induction with Large Language Models that Large Language Models can reverse-engineer game mechanics and build explicit SCMs purely from gameplay traces, significantly improving downstream task performance and interpretability.

Causal and logical verification is equally valuable in traditional tree-search algorithms. For example, Monte Carlo Tree Search (MCTS) evaluates millions of future trajectories, making its final choice opaque to an operator. Ziyan An et al. address this in Enabling MCTS Explainability for Sequential Planning Through Computation Tree Logic by leveraging Computation Tree Logic (CTL) to formally verify these search trees, allowing the system to answer "why," "why not," and "what if" questions in natural language.

Neuro-Symbolic Integration and Hierarchical Abstraction

Rather than extracting explanations after a model is trained, a parallel line of research builds transparency directly into the agent's architecture. This is often achieved by combining the raw optimization power of neural networks with the structural constraints of symbolic logic or hierarchical task design.

Yue Yang et al. propose the Neural+Symbolic Approaches for Interpretable Actor-Critic Reinforcement Learning (NSAC) framework, which pairs a neural "critic" with a rule-ensemble "actor." The neural network handles the complex task of value estimation, while the actor is forced to output human-readable "IF-THEN" logic for its actions. Similarly, the authors of Integrated Commonsense Reasoning and Deep Learning for Transparent Decision Making in Robotics combine Answer Set Programming (ASP) for high-level logical reasoning with Convolutional Neural Networks for low-level perception, enabling robots to justify their beliefs and choices on demand.

Structuring the action space hierarchically provides another layer of intrinsic interpretability. In robotic manipulation, Benjamin Beyret et al. introduce the Dot-to-Dot: Achieving Structured Robotic Manipulation through Hierarchical Reinforcement Learning framework, where a high-level agent plans a sequence of interpretable sub-goals, which a low-level controller then executes. In tactical environments like simulated dogfights, Haoran Han et al. constrain a deep Q-network in Interpretable DRL-based Maneuver Decision of UCAV Dogfight to select from a predefined library of classic flight maneuvers rather than raw control signals, rendering the emergent strategy immediately understandable to human experts.

Post-Hoc Auditing and Introspection

In scenarios where complex neural networks cannot be replaced by symbolic alternatives, researchers deploy post-hoc auditing tools to inspect the agent's internal state or explain specific decisions locally.

For infrastructure control, such as HVAC energy management, Olivera Kotevska et al. combine local feature attribution methods like LIME with global statistical tools like Partial Dependence Plots in Methodology for Interpretable Reinforcement Learning Model for HVAC Energy Control to verify that an agent's decisions align with thermodynamic principles. In semiconductor manufacturing scheduling, Alessandro Immordino et al. combine SHAP values, counterfactual analysis, and decision tree approximations in Explainable AI for reinforcement learning based dynamic scheduling solutions in semiconductor manufacturing to provide multi-layered explanations that bridge the trust gap for production engineers.

Beyond feature attribution, agents can be designed to monitor their own internal representations. Nikos Pitsillos et al. introduce a framework for Intrinsic Robotic Introspection: Learning Internal States From Neuron Activations, using a Variational Autoencoder to compress raw neuron activations into abstract, inspectable internal states. This internal visibility is crucial for safety verification; as Melody Y. Guan et al. demonstrate in Monitoring Monitorability, tracking an agent's internal reasoning paths (such as Chain-of-Thought traces) provides a far stronger safety signal than merely observing its external actions, helping stakeholders detect hidden vulnerabilities like reward hacking before they manifest as failures.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 415 papers
  2. Version 1 · 2026-10-04 · 323 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_black_box_decision_making_9177128c,
  title = {black-box decision making},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-decision-making/rdr_9177128c2286473394d9da6fb89ae5c5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_black_box_decision_making_9177128c_v2,
  title = {black-box decision making},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-decision-making/rdr_9177128c2286473394d9da6fb89ae5c5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake