Pessimism About the Physics, Not the Action: Offline Learning of Short-Packet Schedulers

Radio schedulers for ultra-reliable low-latency links send very short packets, often only a few hundred symbols, and must meet error targets near one in a hundred thousand. We want to learn such schedulers from logs of past network operation, because trial-and-error on a live network drops real packets. The standard offline-learning tool is pessimism: distrust the value of any action that the logs rarely show. That tool fits this problem badly. First, the logs cannot certify reliability. At an error target of $10^{-5}$, a state-action pair needs on the order of $10^5$ logged packets before a single failure is expected. Second, the most useful actions, such as a different blocklength, a different pilot length, or a lower rate, are often ones the logging scheduler never took. Yet the physics of short-packet coding already predicts how they would turn out. The error probability for a given blocklength, rate, and signal-to-noise ratio (SNR) is known in closed form. What is uncertain is how well that formula matches the real receiver.

We propose to place the pessimism on that mismatch. The scheduler's value function uses the analytic finite-blocklength error model to evaluate any action counterfactually on logged states. A small learned correction captures where the real system departs from the model, because of imperfect channel estimates, pilot contamination, or a mismatched decoder. A lower confidence bound is applied only to this correction, and so only in the directions where the logs have not pinned it down. In two sentences: offline schedulers for short packets should trust the coding-theory formula where the data confirm it and be pessimistic where the data cannot. Doing so lets them move beyond the logged actions without claiming reliability the evidence does not support.

Background and Gap

In finite-blocklength (FBL) communication, the achievable rate falls below Shannon capacity by a term that grows as the packet gets shorter and the error target gets stricter. With $C$ the capacity, $V$ the channel dispersion (how much the channel's information density fluctuates), $n$ the blocklength, and $Q^{-1}$ the inverse Gaussian tail function, the rate is approximately $R \approx C - \sqrt{V/n}\,Q^{-1}(\varepsilon) + \log n/(2n)$. This approximation is the basis of Joint Sum Rate and Error Probability Optimization: Finite Blocklength Analysis, which solves a static multi-user rate–error trade-off. It also underlies Energy-Efficient Packet Scheduling With Finite Blocklength Codes, which proves that energy is convex in blocklength and solves deadline-constrained scheduling by water-filling and rolling-window optimization. That paper also motivates this work. Designing short packets with the Shannon formula yields roughly a 50% packet error probability and underestimates energy by about 10% at a target of $5\times10^{-4}$. Both papers assume perfect channel knowledge and known arrivals.

Two further papers constrain the design. A Finite-Blocklength Analysis for URLLC with Massive MIMO and Finite-Blocklength Bounds on the Maximum Coding Rate of Rician Fading Channels show that once pilots eat into the short packet and channel estimates are imperfect, the simple normal approximation stops describing the real error rate. Tighter random-coding-union and saddlepoint bounds are then needed, and the answer depends on the receiver (M-MMSE versus RZF). The physics model is therefore informative but systematically wrong in ways that depend on the deployment. That is exactly the kind of uncertainty a learned correction with a confidence bound can represent.

On the learning side, pessimistic offline reinforcement learning subtracts an uncertainty term from the estimated value, $Q_{\text{pess}} = \hat Q - \beta\Gamma$, as in Conservative Q-Learning (CQL). Offline RL with Differentiable Function Approximation is Provably Efficient supplies the method we reuse. Its pessimistic fitted Q-learning builds $\Gamma$ from the empirical Fisher information of the model's gradients and obtains guarantees that scale with parameter dimension. In both, $\Gamma$ covers the whole value function, and all actions not seen in the logs are penalized alike.

The closest applied work is Offline Reinforcement Learning for Wireless Network Optimization With Mixture Datasets. It runs CQL, BCQ, and IQL on logs from mixed rule-based schedulers and recovers over 95% of a fully trained online agent. Its channel model is Shannon capacity, so decoding errors and short-packet reliability never enter. Risk-aware and deep RL work on URLLC (Risk-Sensitive RL for URLLC Traffic, Reliable RL for Uplink NOMA-URLLC, Experienced Deep RL with GANs for URLLC) trains online in simulators and has no offline confidence bounds.

The obvious combination is to substitute the FBL rate into Yang et al.'s reward and run CQL. It inherits two flaws. It still has to learn from data what the formula already knows, and it still cannot see $10^{-5}$ failures in the logs. We treat that combination as a baseline, not as the contribution.

Core Claim

In short-packet scheduling, the transition structure splits into three parts. The first is exogenous: packet arrivals and channel evolution do not depend on the scheduler's action in a single cell. The second is analytically modeled: decode success given the channel estimate, blocklength, pilot length, power, and rate. The third is deterministic bookkeeping: the queue and age-of-information update given success. Because of this split, a logged exogenous trace can be replayed under any counterfactual action. The only part not identified by the logs is the gap between the analytic success probability and the real one.

We model that gap as a correction to the log-odds of decode failure. Let $p_{\text{FBL}}(s,a)$ be the analytic failure probability from the normal or saddlepoint approximation, evaluated at the estimated SNR. Let $\delta_\theta(s,a)$ be a small learned correction, parameterized as an effective-SNR offset on physical features: estimated SNR, pilot fraction, blocklength, and receiver type. The pessimistic failure probability is

$$\operatorname{logit} p_{\text{pess}}(s,a) = \operatorname{logit} p_{\text{FBL}}(s,a) + \delta_{\hat\theta}(s,a) + \beta\,\Gamma(s,a), \qquad \Gamma(s,a) = \sqrt{g(s,a)^\top \Lambda^{-1} g(s,a)},$$

where $g = \nabla_\theta \delta_\theta(s,a)$ and $\Lambda$ is the Fisher information of the correction fitted on logged success and failure outcomes. This follows the construction of Yin and Wang, applied to $\delta$ rather than to $Q$.

Fitted Q-iteration then backs up the value using $p_{\text{pess}}$, the deterministic queue update, and the logged exogenous next state. Actions whose physics is well explained by the data get a small penalty even if the logging scheduler never chose them. Actions that extrapolate the correction, for example to much shorter pilots or much lower error targets than anything logged, get a large one.

The correction can be learned at all because of a structural fact of the FBL expansion. The error curve is tied to the SNR through its dispersion slope. Logged operating points with observable error rates, around $10^{-2}$ to $10^{-3}$, therefore constrain the SNR offset, and the confidence bound then covers the extrapolation to $10^{-5}$.

We expect three consequences. The learned scheduler should match the model-based optimum when the physics model is exact. It should keep its reliability target when the model is wrong, where naive model-based planning will not. And it should choose blocklength and pilot allocations outside the logged support, where action-level pessimism like CQL will not.

A further practical benefit addresses the known difficulty of tuning $\beta$ without online trials. Because the bound is on a physical quantity, $\beta$ can be chosen offline. We pick the smallest $\beta$ for which the predicted failure bound covers the observed failure rate on held-out logged operating points.

Research Program

We will build a single-cell scheduler first and a multi-cell one second. In the single-cell version, the action chooses blocklength, pilot length, transmit power, and coding rate for each queued packet under a deadline. The reward trades energy against deadline violations and age-of-information exceedances, and the queue dynamics follow the scheduler of Xu et al.

"True" decoding outcomes come from a harder model than the one the learner is given. The ground truth uses random-coding-union or saddlepoint bounds with estimated channels, pilot overhead, and correlated fading, following Lancho et al.'s massive MIMO setup. The learner's anchor uses the perfect-CSI normal approximation. This creates a controlled, tunable model–reality gap, and the gap's size is set by pilot length, mobility, and receiver choice (M-MMSE or RZF).

Logs come from a mixture of behavior schedulers, following the Yang et al. protocol: Random, Greedy, TDM, and ITLinQ adapted to finite blocklength. We add Shannon-designed and margin-backoff convex schedulers, which are the realistic legacy policies an operator would hold.

The analysis has two parts. The first is a finite-sample statement: the backed-up value is a lower bound whenever the correction's confidence set holds. This follows from the PFQL argument with the correction's parameter dimension in place of the Q-network's. The second is an empirical study of where the physics anchor fails. In the multi-cell extension, one cell's power changes another cell's interference, so the transitions are no longer exogenous. There we fold the interference effect into the learned correction and measure how quickly the benefit shrinks.

Baselines

The naive reference point is the best single behavior scheduler in the log mixture, evaluated in the same true environment. Any learned policy that fails to beat it has extracted nothing from the data.

The strongest existing learned method is offline RL on mixture data in the style of Offline Reinforcement Learning for Wireless Network Optimization With Mixture Datasets. There, "Mixed-RL" reached over 95% of a fully trained online RL agent's performance on a Shannon-model scheduling task. No reported result was found on Lacuna for CQL or IQL on any finite-blocklength or URLLC-reliability benchmark, so we will establish these numbers ourselves. We run CQL and IQL twice, once with Shannon rewards (their paper's setting) and once with FBL rewards (the obvious combination). Both runs use identical logs, network sizes, training steps, and the same offline selection budget for $\alpha$ or $\beta$.

The strongest non-learning competitor is the rolling-window convex scheduler of Energy-Efficient Packet Scheduling With Finite Blocklength Codes. We give it the same perfect-CSI normal-approximation model the learner receives and a reliability margin tuned on the logs. We also run model-based dynamic programming, which plans with the analytic model and the empirical arrival process but applies no correction or pessimism.

Two ablations isolate the key ingredient. The first keeps the physics anchor and the learned correction but sets $\beta=0$. The second removes the anchor and applies Fisher-based pessimism to a generic Q-network. Behavioral cloning is included as the imitation reference.

Beating the field means meeting the deadline-violation target, and doing so at lower energy per delivered bit than every baseline that also meets it. The comparison is swept across the model–reality gap. The decisive margin is against FBL-reward CQL and against uncorrected model-based planning.

Why not just…?

Why not just put the FBL rate into the reward and run CQL? This is the natural first attempt, and it may do reasonably well when the logs already cover good blocklength choices. It has two weaknesses. CQL pushes down every unlogged action equally, including those whose outcome the formula predicts reliably. It also estimates failure from logged outcomes that almost never contain $10^{-5}$-level events. The Yang et al. finding that single-policy logs give only modest gains suggests coverage is the binding limit, which is exactly what the physics anchor relaxes. Our experiment settles this by thinning action coverage: dropping all logged packets above a blocklength or below a pilot length. The proposal should degrade gracefully and FBL-CQL should not. If FBL-CQL matches us even in the thinned regime, the anchor is not worth its complexity.

Why not just plan with the analytic model, via convex optimization or dynamic programming? This is the most serious threat. When the formula is right, it should be as good as our method, and we expect a tie at zero mismatch. The Lancho et al. results show the formula is not right once pilots and estimation errors matter. The gap they document between M-MMSE and RZF, where RZF needs 200 antennas to match the reliability M-MMSE achieves with 100, shows how receiver details move the error curve. A planner that trusts the normal approximation will pick operating points that miss the target. The mismatch sweep tests this directly. If model-based planning with a simple tuned margin holds the target across all mismatch levels at comparable energy, our correction adds little, and we would report that.

Why not just back off the error target by a fixed safety factor? A fixed margin is a single global form of pessimism. It overpays where the model is accurate and underpays where it is not. The margin-backoff convex scheduler is in the baseline set to measure this cost directly, as energy spent at equal reliability.

Why not just clone the best logged scheduler, or train online in a simulator? Cloning cannot beat the best behavior policy in a mixture. D4RL (D4RL) and CQL both report that imitation fails when stitching is needed. Online training in a simulator reproduces the simulator's errors, which is the same model-mismatch problem as analytic planning, and exploring on a live network drops real packets.

Proposed Experiments

The first experiment is the single-cell deadline scheduler with Rayleigh block fading, following Xu et al. It uses blocklengths from 100 to 1000 symbols, where Mili et al. report that throughput is most sensitive to codeword length. Error targets range from $10^{-3}$ to $10^{-5}$. The second is a four-cell wrap-around massive MIMO configuration with ten users per cell, following Lancho et al., used as the source of ground-truth outcomes under imperfect CSI. The third adds age-of-information tail reporting, in the style of Abdel-Aziz et al., on the same traces.

Metrics are deadline-violation probability, packet error rate, energy per delivered bit, and the age-of-information exceedance curve. Evaluation uses at least $10^7$ packets per configuration so that $10^{-5}$ rates are resolved.

We will run four decisive checks. First, at zero model mismatch the proposal should tie model-based dynamic programming. Second, as mismatch grows, uncorrected planning should violate the target while the proposal holds it. Third, under thinned action coverage the proposal should beat FBL-reward CQL in energy at equal reliability. Fourth, the pessimistic failure bound should cover the true failure rate on held-out operating points at the nominal confidence level. If the fourth check fails, the certificate is not valid and the main claim fails with it.