A traffic light controller that plans ahead is judged by what it can see. Most learned controllers see queues: how many cars are waiting at the stop line. But a car that has just left the upstream intersection is not yet in any queue. It will arrive in twenty or forty seconds, and it will cost exactly as much as a car already waiting. Changing the light also has a cost of its own, because each change forces a yellow and all-red interval during which nothing moves. We propose a short-lookahead tree search for multi-intersection signal control that handles both facts. The planner scores the end of each simulated plan with a learned value function, which is a network that estimates all future cost from a given state. We define that network on queues plus the cars still in transit, sorted by how soon they will arrive. We also let the planner discard light changes using a relevance test whose time window matches the transit lag, not the next single step.
The two sentences to tell a colleague are these. Short-horizon planners that end their plans with a learned value estimate systematically undervalue congestion that is still travelling toward the intersection, and this bias grows with link length. We correct it by building in-flight arrivals into the value function and into the rule that prunes candidate phase changes. The result should let a planner with a short horizon beat both reactive switching rules and learned reactive policies, and it should do so by the largest margin on long links and at high demand.
Core Claim
Value-augmented planning appends a learned terminal value to a short rollout, as in POLO and TD-MPC. POLO proves that the planner's sensitivity to value error shrinks roughly as $\gamma^H$. That guarantee assumes the value network is fed a state that actually determines the future. In a signalized network with spatial transit delay this assumption fails. Two leaf states with identical queues but different numbers of cars in flight have very different futures. A queue-only critic averages over them and reports the leaf as "clearer" than it is. We call this entry-delay measurement bias, and we claim three things about it. It is measurable as a correlation between the critic's error and the in-flight load at the leaf. It is removed by conditioning the critic on an arrival-time histogram. Once it is removed, the planning horizon no longer needs to span the transit time.
Concretely, the planner at each intersection solves
$$\max_{a_{0:H-1}} \; \sum_{t=0}^{H-1} \gamma^t r_t \;+\; \gamma^H V_\theta(q_H, f_H),$$
where $a_t$ is the signal phase at step $t$ and $r_t$ is the negative squared-queue cost. The yellow and all-red switching interval is modeled inside the rollout as zero discharge. $q_H$ is the vector of lane queues at the leaf. $f_H$ counts vehicles on each incoming link, binned by predicted time to reach the stop line; these predictions come from vehicle position and free-flow speed, refined by the upstream neighbor's announced phase plan. $V_\theta$ is the learned terminal value.
The second ingredient is pruning at each tree node. A candidate phase switch $b$ away from the active phase $c$ is expanded only if
$$\Delta(b,c) = \sum_{k \le \tau_{sw}+g_{\min}} \big[ D_b(k) - D_c(k) \big] > \kappa,$$
where $D_x(k)$ is the demand phase $x$ would serve at step $k$ (queued plus arriving by $k$), $\tau_{sw}$ is the switching interval, $g_{\min}$ is the minimum green, and $\kappa$ is a learned threshold. Keeping the active phase is never pruned. Pruned branches can re-enter through progressive widening, so no switch is discarded permanently.
Relation to Prior Work
Three pieces of work motivate the gap. Chen and Cassandras model inter-intersection transit delay in stochastic fluid models and optimize cycle lengths by online gradients. They report a roughly 40% reduction in squared-queue cost from delay awareness, with the advantage growing with link length. They also report almost no difference under average-queue cost. That last observation tells us which metric exposes the bias. Their method tunes continuous cycle splits, allows at most one platoon per link, and has no terminal value. Han et al. show that continuum models overstate effective green time because they ignore start-up and clearance lost time. This is why we simulate switching loss inside the rollout instead of adding it as a penalty. Hsieh et al. prove that plain Max-Pressure collapses under switch-over delay. They fix this with Biased Max-Pressure (B-MP), a constant bias toward the active phase. B-MP is reactive and cannot open a phase for a platoon it cannot yet see. Our threshold $\Delta$ can be read as a B-MP bias that looks forward over the lag window.
The closest existing method is Delayed EfficientZero. It handles execution delay by rolling a learned model forward over pending actions and starting tree search from the anticipated "future-present" state. It also realigns replay data. Our correction is similar in spirit: it completes the state before the value network judges it. The differences are three. DEZ addresses the agent's own action lag, whereas we address transit lag of exogenous load that is still to be counted. DEZ keeps the full branching factor. DEZ assumes an observable delay and has no notion of zero-throughput switching intervals.
The pruning side builds on and departs from three methods. On-the-fly state-conditioned action abstraction merges sub-actions whose one-step effect on the next latent state is indistinguishable. It cuts search by up to 95.2% in Factored Sokoban. Under transit lag, however, a one-step test labels a switch irrelevant exactly when its effect is delayed, so we widen the test window to the lag. DG-MCTS prunes by least-laxity urgency against static deadlines. Smith et al. show that forward pruning is safe when sibling values are correlated. That analysis constrains us: we must check that the pruned siblings are in fact low-value, not merely delayed.
One cited result appears to contradict the proposal. LIT argues from Webster's formula that queue counts suffice. It also finds that adding vehicle positions to the state slowed learning. We take this seriously. Our in-flight feature is a coarse histogram of a few bins, not raw positions. LIT's equivalence assumes steady flow within each segment, and platoons on long links violate exactly that. The ablations below test whether LIT's finding holds in our setting.
Research Program
The work has three parts, built in order. The first is diagnostic. We train a queue-only critic in the standard TD-MPC/POLO style on CityFlow rollouts. We then regress its error against realized return-to-go on the in-flight load at the leaf, and sweep link length on a synthetic corridor. If the error does not grow with in-flight load and link length, the premise fails and we stop.
The second part is the planner. Each intersection runs Monte Carlo tree search over its phases. It uses the simulator's own dynamics for the short rollout: this is simpler than a learned model and avoids compounding model error. Neighbors exchange their best plans once per decision, so that downstream arrival forecasts $f$ reflect upstream discharge. The value $V_\theta$ is trained offline on multi-step targets generated by the planner, following POLO's plan-online, learn-offline loop.
The third part is the pruning rule. We fit $\kappa$ jointly with the value and log every pruned branch. Offline, we re-expand a sample of pruned branches to estimate how often pruning discards the best action.
Baselines
The naive reference points are fixed-time control and plain Max-Pressure, both run with the same yellow and all-red intervals as every other method. LIT reports fixed-time at 682.50 s average travel time in its Los Angeles SUMO setting; that figure is only a scale reference, since we rerun everything in CityFlow.
The strongest rule-based competitor is B-MP from Hsieh et al., which reports over 40% lower average delay than a perfectly tuned fixed-time plan at 2600 veh/hr in VISSIM. Its bias parameter $\alpha$ is the paper's own acknowledged weak point. We therefore give it a grid search per network, with the same tuning budget as our $\kappa$ and $H$.
The strongest learned competitor is HiLight, a hierarchical reactive policy with local and neighborhood critics. It reports 223 s average travel time on the Shenzhen weekend network, against 1099 s for CoLight and 1444 s for PressLight, and throughput of 5769 vehicles on Jinan. We rerun HiLight and FRAP from their released configurations. All methods use the same demand files and switching intervals, and the same number of environment steps for training. We also match per-decision wall-clock across all planners.
The planning baseline is a DEZ-style tree search: full branching, queue-only value, same rollout budget. It isolates the effect of the two corrections from the effect of planning itself. Two ablations do the rest of the isolation: our planner with a queue-only value, and our planner with one-step pruning in place of the lag-window pruning.
Beating the field means lower average travel time than HiLight and B-MP on Shenzhen and Jinan with non-overlapping seed intervals. It also means a squared-queue cost gap over B-MP that grows with link length on the synthetic corridor.
Why not just…?
Why not just tune Biased Max-Pressure? B-MP is throughput-optimal and cheap, and at moderate demand a well-tuned bias may match any planner. Its limitation is structural: it reacts only to queues already formed. Chen and Cassandras show that failing to anticipate platoons costs about 40% in squared-queue cost on long links. Our link-length sweep settles the question. If tuned B-MP stays within noise of our planner even at large link length, the lookahead is not earning its cost and we would report that.
Why not just add in-flight counts to a model-free policy's state? This is the real threat. It is possible that most of the gain comes from the feature and not from planning. The ablation that answers this is a FRAP/HiLight variant given the same arrival histogram $f$. If it closes most of the gap, the contribution shrinks to a state-design lesson, still testing LIT's claim. What planning adds is explicit reasoning about switching lost time and neighbor plans. The prediction is that the advantage appears specifically in high-demand regimes where switching decisions dominate.
Why not just lengthen the horizon to cover the transit time? POLO's $\gamma^H$ bound suggests that a longer horizon washes out value error. But branching grows exponentially with depth, and lags of tens of seconds mean deep trees. We compare a queue-only planner at horizon $H$ equal to the transit time against our planner at short $H$, with matched compute. We expect equal quality at a fraction of the decision time.
Why not just use DEZ unchanged? DEZ corrects the agent's own action lag and does not count exogenous load in transit. It also expands every phase at every node. The DEZ-style baseline above measures both shortcomings directly.
Proposed Experiments
We use CityFlow (Zhang et al.) with the Jinan, Hangzhou, and Shenzhen datasets. We add a synthetic two-intersection corridor that sweeps link length and demand, mirroring Chen and Cassandras. Metrics are average travel time, throughput, squared-queue cost, fraction of time spent in switching intervals, per-decision latency, and pruning error rate (how often a re-expanded pruned branch beats the chosen action).
Four checks are decisive. First, the queue-only critic's error must correlate with leaf in-flight load, and our critic must remove that correlation. Second, the planner's gain over B-MP and HiLight must grow with link length and demand. Third, lag-window pruning must reduce nodes expanded while keeping pruning error below that of one-step pruning, which we expect to prune delayed-but-good switches. Fourth, the feature-augmented reactive baseline must fall measurably short in high-demand settings. If this fourth check fails, we report the result as primarily about state design rather than planning.
References
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
- Temporal Difference Learning for Model Predictive Control
- Stochastic Flow Models with Delays and Applications to Multi-Intersection Traffic Light Control
- On the continuum approximation of the on-and-off signal control on dynamic traffic networks
- Throughput-Optimal Scheduling for Multi-Hop Networked Transportation Systems With Switch-Over Delay
- Tree Search-Based Policy Optimization under Stochastic Execution Delay
- Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction
- Online Decision-Making Under Uncertainty for Vehicle-to-Building Systems
- An Analysis of Forward Pruning
- Diagnosing Reinforcement Learning for Traffic Signal Control
- Hierarchically and Cooperatively Learning Traffic Signal Control
- Learning Phase Competition for Traffic Signal Control
- CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario