Distilling Deep Reinforcement Learning into Symbolic Policies

461 concepts310 papers2 proposals

Deep reinforcement learning (DRL) is exceptionally good at finding high-performing control policies in complex, high-dimensional spaces. However, the resulting behavior is typically locked inside an opaque black box of millions of continuous neural network weights. This creates a fundamental challenge: we have agents that can solve tasks perfectly, but we cannot easily explain why they make specific decisions, formally verify their safety before deployment, or run them on resource-constrained hardware.

To bridge this gap, a powerful paradigm has emerged: Symbolic Policy Distillation. Instead of forcing humans to trust a black box, researchers use the trained neural network as a guide to discover a compact, human-readable symbolic policy—such as a decision tree, a programmatic script, or a set of logical rules.

The Core Mechanism: Oracle-Guided Synthesis

Why not just learn a symbolic policy from scratch? The space of discrete programs, decision trees, or logical rules is combinatorially explosive, non-differentiable, and highly sensitive to sparse rewards. Searching this space directly using evolutionary algorithms or brute-force enumeration is often computationally intractable for complex tasks.

Symbolic policy distillation bypasses this bottleneck by treating a pre-trained neural network as an oracle or teacher. As surveyed by Glanois et al., this process leverages imitation learning (or mimic learning) to transfer knowledge from the continuous domain to the discrete domain through a simple two-step workflow:

  1. Generate Demonstrations: The high-capacity neural network interacts with the environment, mapping out the state space and generating a rich dataset of state-action trajectories.
  2. Synthesize the Structure: A symbolic synthesis engine uses these trajectories to find a compact representation within a Domain-Specific Language (DSL) or rule grammar that best mimics the oracle's behavior.
[ Environment ] <--- Actions --- [ Deep Neural Policy (Oracle) ]
       |                                      |
       |--- States & Trajectories ------------|
       v
[ Trajectory Dataset ] ---> [ Synthesis Engine ] ---> [ Compact Symbolic Policy ]

For example, Verma et al. introduced Programmatically Interpretable Reinforcement Learning (PIRL), which uses a neural policy to smoothly navigate the environment's reward landscape and then distills that behavior into clean, programmatic code. Similarly, Eberhardinger et al. demonstrated that complex multi-agent policies can be distilled into an ensemble of logical programs, converting high-dimensional neural representations into human-readable code. By using the neural network to solve the hard exploration problem first, the symbolic synthesizer only needs to solve a supervised regression or classification problem.

The Flavors of Symbolic Representation

Depending on the deployment constraints and the domain, the target symbolic structure can take several forms, each offering unique intuitive advantages:

  • Decision Trees and If-Then Rules: Complex continuous decision boundaries are broken down into hierarchical conditional logic. Liu et al. utilized Linear Model U-Trees to approximate deep policies, allowing developers to extract explicit if-then rules and calculate exact feature influences. Roth et al. showed that distilling a robot navigation policy into a decision tree allows engineers to pinpoint and manually correct specific failure modes (like oscillatory behavior) by editing individual nodes directly, rather than guessing a new reward function and retraining from scratch.
  • Programmatic Code: Representing policies as executable scripts allows the integration of classical control primitives (like PID loops). Gu et al. synthesized programmatic traffic signal controllers using Monte Carlo Tree Search, producing human-readable scripts lightweight enough to run natively on low-power 8-bit microcontrollers with minimal memory.
  • First-Order and Relational Logic: Instead of binding logic to specific state coordinates, relational rules describe interactions between abstract objects. Jiang et al. and Hazra demonstrated that expressing policies via first-order logic provides inherent interpretability, creating a verifiable reasoning trace that explains an agent's actions in human-readable terms.

Why Symbolic Policies Generalize Better

Intuitively, neural networks are prone to overfitting to the specific geometry, coordinate systems, and object counts of their training environments. When a neural agent learns to navigate a grid or sort blocks, it often memorizes high-dimensional correlations between raw input features and actions.

Symbolic distillation forces an informational bottleneck. Because a program or a shallow decision tree has strictly limited capacity, it cannot memorize the dataset; it is forced to compress the policy into generalized principles.

As shown by Eberhardinger et al., mapping neural trajectories to a domain-specific language extracts reusable symbolic concepts. A distilled rule like if obstacle_ahead: turn_left is inherently invariant to the size of the map or the exact coordinate values. As a result, these distilled policies routinely achieve superior extrapolation and "zero-shot" generalization when dropped into larger, unseen environments where neural networks completely break down.

The Performance-Interpretability Trade-off

While symbolic distillation offers massive benefits in terms of auditability, verification, and computational efficiency, it introduces a fundamental trade-off. Eberhardinger et al. note that programmatic policies frequently suffer from a performance gap compared to their neural teachers, failing to reach the same peak reward on the original training task. The discrete, rigid nature of symbolic logic struggles to capture the hyper-fine, continuous adjustments that neural networks naturally excel at.

To mitigate this, modern frameworks move away from purely decoupled synthesis. For instance, the DisCo-DSO framework proposed by Pettit et al. jointly optimizes the discrete structure of a policy alongside its continuous thresholds. By simultaneously evolving the logic tree and tuning its numerical parameters, these hybrid approaches narrow the performance gap, proving that highly interpretable, distilled models can remain competitive with black-box neural networks.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 315 papers
  2. Version 1 · 2026-10-04 · 364 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_symbolic_policy_distillation_2897631f,
  title = {symbolic policy distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-policy-distillation/rdr_2897631fea90420da0839e3b65edee02},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_symbolic_policy_distillation_2897631f_v2,
  title = {symbolic policy distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-policy-distillation/rdr_2897631fea90420da0839e3b65edee02/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake