Counterfactual Replays as Unit Tests: LLM Refinement of Wireless Scheduling Programs

Deep reinforcement learning is the default tool for allocating radio resources, such as deciding which base station serves each phone and how much power or bandwidth each link gets. The policies it produces are neural networks. They are hard to inspect, costly to run on cheap radio hardware, and prone to breaking when the network layout changes. Large language models can write and repair code when a compiler or test suite tells them what went wrong. That suggests a different route: have the language model write the scheduler itself as a short program, run it in a network simulator, and refine it. The trouble is that a network simulator gives no failing test and no stack trace. It returns a reward curve. A language model told only that "return dropped from 1.62 to 1.55" has little to reason with, which may be one reason refinement loops plateau after a few rounds.

This proposal turns simulator episodes into something that looks like failing unit tests. After each episode, we clone the simulator at chosen time steps and try alternative actions for a short horizon. We then hand the model the handful of concrete moments where its program's decision was clearly worse than an available alternative, together with the state it saw and what would have happened otherwise. We call these moments counterfactual replays. The claim is that feeding them back, and accepting new program versions by their robustness across a pool of perturbed network scenarios rather than by gains on the training seeds, yields compact scheduling programs. These programs should match neural policies on the training scenario and beat them when the scenario shifts. It should also do so at lower language-model cost than refinement that sees only the scalar score.

Core Claim

Existing work splits along a clear seam. Code refinement systems repair programs against deterministic oracles. ICER (Xie et al., 2026) refines runtime efficiency against unit tests. POLO (Bai et al., 2025) ranks bottlenecks in multi-file C++ projects. LLM-Vectorizer (Taneja et al., 2024) checks vectorized loops with a formal equivalence checker. None of these faces a stochastic, closed-loop environment. LLM-driven algorithm design, as in $A_2$DEPT (ICML 2026) and EoH-S (Liu et al., 2026), does evolve resource-allocation and scheduling code, but for offline batch instances scored by a single fitness number. The network-side LLM work is also narrower than it first appears. Rezazadeh et al., 2025 repairs ns-3 simulation scripts rather than the scheduling logic, and Du et al., 2025 uses an LLM only as a judge of user experience inside a neural actor-critic.

Our specific claim is that the missing ingredient for reactive control is localized, counterfactual diagnostic feedback. Scalar-fitness evolution cannot use such feedback, and conventional refinement assumes it already exists. Given a program policy $\pi$ and a state $s_t$ visited during an episode, we estimate the local regret of its decision as

$$ \delta_t = \max_{a \in \mathcal{A}_t} \hat{G}_H(s_t, a) - \hat{G}_H(s_t, \pi(s_t)). $$

Here $\mathcal{A}_t$ is a small set of candidate actions: the program's own action, single-user association changes, and actions from a reference heuristic. $\hat{G}_H(s_t,a)$ is the average reward over the next $H$ steps obtained by cloning the simulator at $t$, taking $a$, and then following $\pi$, averaged over a few noise seeds. The top-$k$ moments by $\delta_t$ become the "failing tests" shown to the model. Each is presented as the observed state, the program's action, the better action, the reward difference, and the code path the program took.

We also claim that the acceptance rule matters as much as the feedback. Kalntis et al., 2025 show that virtualized base-station environments are non-stationary to the point of being adversarial. ICER separately finds that strict per-step improvement on a small evaluation set makes refined code overfit (Xie et al., 2026). We therefore adopt ICER's relaxed acceptance idea and move it from runtime efficiency to control. A new version is kept if it does not lose on a rotating pool of perturbed scenarios, even if it does not improve on the training seeds.

Research Program

The environment is mobile-env, an open-source Gym simulator for coordinating users across base stations. Its standard setting has 3 stations, 3 moving users, and a 13-element observation of signal quality and station load. It is attractive because Nowosadko et al., 2026 report neural, self-explaining, and heuristic returns on it, which gives us fixed reference points.

The program interface is a Python function from observation to action. It may keep a small bounded state object between calls, which lets it express hysteresis, timers, and per-user memory. Neural policies and the explanations of Nowosadko et al., 2026 are blind to this kind of multi-step behavior. The program must pass static checks before any episode runs: no imports beyond a whitelist, bounded loops, and a fixed memory budget.

The loop starts from the dynamic-selection heuristic. Each round then runs training episodes, mines counterfactual replays, builds a prompt containing the current program, its per-scenario returns, and the top-$k$ replays, and asks the model for a revised program. A small population of the best-scoring programs is kept, as in A_2DEPT, so the comparison against scalar evolution changes only the feedback and not the search structure. We also run the dependency and consistency check that $A_2$DEPT introduces, because the "partial migration" failure it documents, where one code block changes a data structure and another does not, is likely when programs start carrying state.

Three analyses go beyond headline returns. The first measures how many rounds pass before improvement stalls. Yan et al. report that refinement gains usually vanish after 2 to 4 iterations, and we test whether replays push that point later. The second measures what fraction of accepted edits touch the code path named in a replay, which tells us whether the model is actually using the diagnostics. The third measures cost, counted as simulator steps spent on cloning plus language-model tokens, and charges it against our method in every comparison.

A secondary domain is CityFlow traffic-signal control, because π-Light (Gu et al., 2024) established there that small programmatic policies can beat neural ones and run on 8-bit microcontrollers. This lets us test whether the method transfers beyond one wireless simulator.

Baselines

The naive reference point is the dynamic-selection heuristic in mobile-env. Nowosadko et al., 2026 report a mean return of 1.428 for it, and it is also our starting program, so any gain is measured from there.

The strongest existing method on the in-distribution task is a PPO neural policy. Nowosadko et al., 2026 report a mean return of 1.803 for a black-box network and 1.805 for their self-explaining variant. We retrain both with their published setup and give PPO a tuning budget at least equal to our method's total simulator steps. Our method's step count includes all cloning rollouts, so the comparison in environment interactions is not tilted in our favor. On the standard scenario we count parity as a mean return within one seed standard deviation of 1.80. "Beating" PPO means a statistically higher return on the shifted scenarios while holding that parity.

The strongest competitor on the actual question, whether diagnostic feedback helps LLM program search, is scalar-fitness LLM evolution in the style of EoH-S (Liu et al., 2026) and $A_2$DEPT (ICML 2026). It uses the same model, the same population, and the same number of language-model calls, and the prompt carries per-scenario returns but no replays. Because replay mining consumes extra simulator steps, we also run a variant of this baseline that spends the equivalent steps on more evaluation seeds. No reported result was found on Lacuna for LLM heuristic evolution on mobile-env. The file's figures come from other benchmarks: $A_2$DEPT reports a 9.8% reduction in mean normalized optimality gap on FrontierCO, and EoH-S reports a 43.6% improvement on TSPLib over prior LLM heuristic design. We beat this baseline if we reach higher shifted-scenario return at equal calls and equal simulator steps, or the same return with fewer calls.

The grammar-based program search baseline is π-Light-style Monte Carlo tree search with Bayesian optimization of constants over a DSL built from the mobile-env observation fields (Gu et al., 2024), run with the same simulator budget. No reported result was found on Lacuna for this method on mobile-env. On CityFlow, the paper reports that it outperforms neural baselines in five of six road networks, and we compare against it there directly.

Ablations isolate each ingredient. One removes the replays but keeps everything else. One replaces counterfactual replays with the worst-reward time steps, which gives localization without a counterfactual. One swaps relaxed acceptance for strict monotonic acceptance on training seeds. A one-shot generation baseline shows how much the loop itself contributes.

Why not just…?

Why not just deploy PPO, which already scores about 1.80? On the training scenario PPO is a real threat, and we do not expect to beat it there. Our case rests on two other axes. The first is behavior under changed user counts, layouts, and speeds, where the file notes that deep policies degrade sharply. The second is deployability. Kalntis et al., 2025 stress sub-millisecond decision budgets, and Gu et al., 2024 show that programs fit on microcontrollers where networks do not. If PPO matches our programs under shift, the robustness claim fails and only the footprint argument remains. The shift experiment settles this.

Why not just distill the trained PPO policy into a decision tree or rule list? This is a cheap and credible alternative, and we include it as an extra baseline with the same program-size limits. Distillation copies the teacher's behavior, including its failures under shift, and it cannot add memory the teacher lacks. The explanation work of Nowosadko et al., 2026 notes that per-step explanations cannot capture temporal strategy. If distilled trees match our shifted-scenario returns, the language model is unnecessary for this domain, and we would report that.

Why not just run scalar-fitness LLM evolution with more samples? This is the closest threat. FunSearch-style population search works well on combinatorial problems (Liu et al., 2026). The equal-call and equal-step comparisons above are designed to decide the question. If the gap closes once scalar evolution gets the extra seeds, our contribution shrinks to the acceptance rule.

Why not just search a DSL with tree search, as π-Light does? A DSL cannot express per-user timers or bounded memory unless someone designs them in by hand. The $A_2$DEPT authors argue that grammar-bound search cannot invent new subroutines (A_2DEPT). If a DSL that includes a hysteresis primitive matches us, then the benefit comes from domain knowledge, not from the language model. We test exactly that variant.

Proposed Experiments

The primary experiment trains every method on the standard 3-station, 3-user mobile-env scenario. We then evaluate on shifted configurations with more users, moved stations, and faster users, none of which are seen during training. Each method runs with 10 seeds, and we report mean return with confidence intervals. We also report program length, per-decision latency on a microcontroller-class CPU, language-model tokens, and total simulator steps.

The decisive check is the replay ablation at matched cost. If full replays do not raise shifted-scenario return over both scalar evolution and worst-step localization, the core claim is false. The second check asks whether strict acceptance widens the gap between training return and shifted return, as ICER's overfitting finding predicts. The third check asks whether the round at which improvement plateaus moves later than the 2 to 4 iterations reported by Yan et al..

Finally, we repeat the replay-versus-scalar comparison on CityFlow against π-Light, to test whether the effect carries beyond a single wireless simulator.