Discovering Programmatic Policies from Reinforcement Learning-Based Traffic Signal Controllers

Lindong Xie, Yang Zhang, Beiyu Song, Zeren Xing, Flora D. Salim, Edward ChungarXiv 2026 · arXiv

TL;DR

Researchers have developed PPL-TSC, a framework that uses Large Language Models (LLMs) to "distill" complex, black-box Reinforcement Learning traffic controllers into compact, human-readable mathematical formulas. These programmatic policies achieve competitive performance while reducing computational costs by up to three orders of magnitude and generalizing far better to new cities.

What Problem Does This Paper Address?

Deep Reinforcement Learning (DRL) has become the gold standard for adaptive traffic signal control (TSC), outperforming traditional fixed-time lights by reacting to real-time sensor data. However, DRL has three major "deployment deal-breakers":

  1. Opacity: Neural networks are black boxes. It is nearly impossible for a traffic engineer to inspect a million-parameter model and verify that it follows safety principles (like "don't ignore a massive queue on a side street").
  2. Resource Intensity: DRL models require significant memory and floating-point operations (FLOPs). Most existing traffic light controllers run on low-power microcontrollers that simply can't handle heavy neural inference.
  3. Brittleness: A DRL model trained on the grid of Manhattan often fails miserably when moved to the irregular intersections of Los Angeles because it has "overfit" to specific network topologies.

The authors aim to bridge the gap between the high performance of DRL and the transparency/efficiency of classical rule-based systems.

Methods

The authors propose PPL-TSC (Programmatic Policy Learning for TSC), which treats a pre-trained DRL model as a "teacher" and attempts to evolve a "student" policy represented as a Phase Utility Function (PUF).

A PUF is a simple mathematical expression (e.g., $Utility = Pressure + 0.2 \times Active\_Phase\_Bonus$) that scores different signal phases. The phase with the highest score gets the green light.

The framework uses a four-step evolutionary loop:

  • Generation: Instead of random mutations used in traditional genetic programming, an LLM acts as the evolutionary operator. It views the teacher's data and proposes Python-based utility functions, using its semantic understanding of traffic concepts (like "queue length" or "delay") to suggest sensible rules.
  • Review: An automated "Reviewer" checks the code for syntactic errors, mathematical stability (no dividing by zero), and "TSC principles"—for instance, ensuring that increasing traffic demand doesn't result in a lower priority for that lane.
  • Calibration: Since the evolved rule is shared across all intersections, the system uses CMA-ES (a derivative-free optimizer) to fine-tune small, intersection-specific "bias" terms to account for local geometry.
  • Evaluation: Candidates are ranked based on a joint score of how well they mimic the teacher (fidelity) and how well they actually reduce travel time in a simulator.

Datasets

The study utilizes the CityFlow microscopic traffic simulator across several real-world road networks:

  • Hangzhou, China: Three different 1x1 intersection scenarios.
  • Atlanta, USA: A 5x1 corridor.
  • Jinan, China: A 3x4 grid.
  • Los Angeles, USA: A 4x4 grid.
  • Manhattan, USA: A large 16x3 grid.

The researchers tested PPL-TSC against five representative DRL "teacher" families: CoLight, FRAP, MPLight, EcoLight, and TinyLight.

Key Insights and Ideas

  • LLMs as Semantic Searchers: The "aha moment" here is using the LLM not to write a final policy, but to act as a "smart" mutation engine. Traditional genetic programming often generates "garbage" math; LLMs understand that "pressure" and "queue" are related, leading to much faster discovery of meaningful control logic.
  • The Power of Symbolic Distillation: The paper demonstrates that the complex "logic" inside a massive neural network can often be compressed into a few lines of math without losing performance. In some cases, the distilled student actually outperformed the teacher by smoothing out the teacher's overfitted quirks.
  • Generalization via Principles: Because the evolved programs rely on universal traffic features (like "pressure") rather than specific neural weights, they are naturally more robust. When transferred from Manhattan to LA, the programmatic policies maintained their performance while the DRL teachers saw significant degradation.

Key Results

  • Extreme Efficiency: For the FRAP model, storage requirements dropped from 5,476 bytes to just 47.2 bytes. Computational cost plummeted from 183,676 FLOPs to a mere 80. This allows these advanced policies to run on a $2 microcontroller like the ATmega328P.
  • Competitive Performance: PPL-TSC achieved positive relative improvements in Average Travel Time (ATT) for most teacher families, with gains as high as 22.7% in specific scenarios.
  • Superior Transfer: In cross-scenario transfers (e.g., training in one city and testing in another), PPL-TSC students outperformed their DRL teachers by 33% to 54% on average.

Limitations and Future Work

  • Teacher Ceiling: The student is guided by the teacher's rollout data. If the DRL teacher has "blind spots" in its training data, the student may inherit those weaknesses.
  • Fixed Policy Class: The current framework focuses on Phase Utility Functions. It might struggle to distill DRL models that rely on complex temporal dependencies or multi-agent communication that can't be expressed in a simple per-phase score.
  • LLM Dependency: The quality of the search is sensitive to the LLM backbone. The authors found Gemini 3.1 Flash performed best, while smaller or older models were less effective at generating valid, high-performing code.

Future work aims to extend PPL-TSC to even larger-scale deployments and explore "closed-loop" optimization where the student continues to learn from the environment independently of the teacher.

Summary updated

Abstract

Traffic signal control (TSC) is critical for mitigating urban congestion. Deep reinforcement learning (DRL) has achieved strong performance by learning adaptive policies from traffic observations, but learned neural controllers are often opaque, structurally complex, and difficult to generalize across scenarios. We propose PPL-TSC, a programmatic policy learning framework that distills compact and human-readable TSC policies from DRL teachers. PPL-TSC formulates policy learning as an evolutionary search over phase utility functions (PUFs), which score candidate signal phases and select the highest-utility phase for execution. Each generation consists of four steps: policy generation, policy review, policy calibration, and policy evaluation. In the generation step, a large language model (LLM) acts as a semantics-aware evolutionary operator to propose and refine candidate PUFs. Generated PUFs are screened through explicit checks for syntactic validity, mathematical soundness, and consistency with TSC principles, with invalid ones repaired through iterative feedback. Valid PUFs are then calibrated to improve behavioral fidelity to the DRL teacher, and evaluated using a joint score combining teacher fidelity and traffic-control performance to select candidates for the next generation. Extensive experiments across diverse traffic scenarios and representative DRL-based controllers show that PPL-TSC produces compact and human-readable policies, achieves competitive performance for most teacher families, improves transfer generalization, and substantially reduces computational cost. Code is available at https://github.com/ForrestXie9/PPL-TSC.
BibTeX
@misc{xie2026discovering,
  author = {Lindong Xie and Yang Zhang and Beiyu Song and Zeren Xing and Flora D. Salim and Edward Chung},
  title = {{Discovering Programmatic Policies from Reinforcement Learning-Based Traffic Signal Controllers}},
  year = {2026},
  eprint = {2611.19951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19951},
}
Figure