Millimeter-wave radios, which operate at 30–100 GHz, send energy in narrow beams that behave almost like light. When a pedestrian or truck steps into the path, the received signal can drop by 20 to 40 dB at once. Today's options are either to sweep many candidate beam directions, which is safe but costs airtime, or to predict the best beam from cameras, LiDAR, and GPS, which is fast but trained on clean line-of-sight data and fails in ways it cannot detect. We propose to train a beam controller only from logs that a base station already records. At each step the controller decides whether to commit to a predicted beam, probe a few candidates, or run a full sweep. It is trained to protect the bad tail of delivered throughput, not just the average. Because it never explores on a live network, it is kept pessimistic about any action the logs never tried.
In two sentences: deployed beam management already produces logs, but those logs only show the power of the beams that were actually probed, so a learner that trusts its own guesses about unprobed beams will be overconfident exactly when blockage or sensor degradation pushes it off the training data. We combine a pessimistic, distribution-aware critic with a sweep-or-commit action space, and we test on real multimodal vehicular data where every beam's true power is known. This lets us measure exactly how much outage that overconfidence causes and how much pessimism removes.
Core Claim
A controller trained offline with conservative quantile value estimation will match the mean effective throughput of the best supervised multimodal beam predictors while substantially reducing outage and the lower tail of throughput. Effective throughput here means rate after subtracting sweep airtime. The tail gains should appear under three kinds of shift: day-to-night, infrastructure-to-vehicle links, and injected blockage. The value comes specifically from two pieces together: pessimism about unlogged actions and optimizing a tail-risk objective. Neither a scalar conservative critic nor an unconstrained distributional critic should achieve it alone. If a simpler confidence-triggered sweep rule does just as well, we will say so. That outcome would itself be the finding: sequential offline RL is unnecessary for this problem.
Where This Sits
Three prior systems come close, and each leaves out a different piece.
Millimeter Wave Communications with an Intelligent Reflector jointly optimizes base-station precoding and reflector phases around a blocked path. It learns a quantile-regression distributional RL policy under imperfect channel knowledge and reports a 10% rate gain over Q-learning. However, it learns online by interaction and still optimizes the expected rate. We keep its distributional critic but drop live exploration and target the tail.
Decision Transformer for IRS-Assisted Systems is offline. It pre-trains a return-conditioned sequence model on many synthetic Rician channel distributions and adapts with few-shot prompts, converging 3 times faster than PPO retraining. It has no mechanism that stops the model from producing actions for returns the data never supported, and it is evaluated only on synthetic single-user channels.
Efficient Beam Selection for ISAC in Cell-Free Massive MIMO applies Conservative Q-Learning to beam selection on digital-twin data. It penalizes only scalar values, does not model the return distribution, and has no blockage or sensor shift.
The method we reuse is Conservative Offline Distributional Reinforcement Learning (CODAC). CODAC pushes the predicted return quantiles down for actions outside the data and then optimizes Conditional Value-at-Risk, which is the average return over the worst $\beta$ fraction of outcomes. It was the only method with zero violations on a risky navigation task.
The motivation comes from two multimodal beam predictors on DeepSense 6G: Multi-Modality Sensing in mmWave Beamforming and When VLM Meets Beam Prediction. Both are strong under line-of-sight conditions. The first paper's authors state that the non-line-of-sight case is future work, and the second paper's authors leave generalization to unseen environments open.
Two papers constrain the design. The first is An Explainable Failure Prediction Framework, which found that radio link indicators carried about 86% of failure-prediction importance and that weather inputs were nearly useless. This tells us to give the controller recent received-power history and to treat sensor features as optional, testing whether they help rather than assuming it. The second is The Surprising Difficulty of Search in Model-Based RL, which shows that planning over a learned channel model exploits that model's errors. This is why we use a model-free critic instead of rolling out a learned blockage simulator.
Research Program
The benchmark. The central construction is a logged-feedback benchmark built from DeepSense 6G scenarios 31–34 (vehicle-to-infrastructure) and 36–39 (vehicle-to-vehicle). These scenarios record the received power of every beam in the codebook at every time step, alongside camera, LiDAR, and GPS.
To produce realistic training logs, we run behavior policies over these sequences and reveal only the powers of the beams each policy actually probed. The behavior policies are a hierarchical sweep with beam tracking and a supervised predictor with occasional sweeps. The learner sees only these partial logs. The full power vectors are held back as ground truth, so any learned policy can be scored counterfactually without online rollouts. This turns a dataset collected for supervised beam prediction into an exactly evaluable offline control benchmark.
Blockage. The file does not establish that these scenarios contain blockage events. We therefore inject blockage: a stochastic blocker process applies a 20–40 dB drop to the dominant beam cluster for random durations, the range reported for mmWave obstruction. The controller must learn that a non-dominant reflected beam is the safer choice at that moment.
Multi-antenna extension. As a second testbed, we use the DeepMIMO O1 60 GHz ray-traced scenario. We remove line-of-sight rays to create blockage and add a simple reflector, which enlarges the action space to beam-and-reflector-configuration pairs. This is where the cost of multi-antenna optimization becomes a real constraint. The reflector paper's estimation overhead grows with the element count $N$, so the sweep-or-commit decision directly trades airtime against information.
The controller. At each step $t$ the agent observes a state $s_t$: the last few measured beam powers, the time since the last sweep, and optional sensor embeddings taken from a frozen multimodal predictor. It chooses an action $a_t$ from three kinds: commit to one of the top-$k$ predicted beams, probe a small local set, or run a full sweep.
The reward is the effective rate
$$r_t = (1 - \rho(a_t))\,\log_2(1 + \mathrm{SNR}_t(a_t)),$$
where $\rho(a_t)$ is the fraction of the slot spent probing and $\mathrm{SNR}_t(a_t)$ is the signal-to-noise ratio on the beam finally used.
We train CODAC's conservative quantile critic on the logs and act to maximize CVaR at level $\beta$ of the discounted return. The conservatism weight $\alpha$ is the known weak point, since CODAC's authors note that it needs tuning and live validation is unavailable. We select $\alpha$ with an off-policy estimate computed from logged data only. We also report the result under an oracle-selected $\alpha$, which shows how much performance this practical restriction costs.
Baselines
Naive references. The do-nothing reference is the logged behavior policy itself: hierarchical codebook sweeping with tracking, which is close to what 5G-NR does. It sets the overhead and outage levels any learned method must improve on. Full exhaustive sweeping every slot gives the best possible beam with maximum overhead, and it serves as the reference for the throughput–overhead trade-off.
Strongest competitor. The strongest competitor is a supervised multimodal predictor combined with a confidence-triggered fallback sweep. We use the GPS, camera, and LiDAR fusion model of Multi-Modality Sensing in mmWave Beamforming, with its softmax confidence thresholded to decide when to sweep, and the threshold tuned on the same logs. On clean line-of-sight data this predictor reports a 99.92% received-power ratio for top-1 prediction on vehicle-to-infrastructure scenarios and 98.19% top-13 accuracy. When VLM Meets Beam Prediction reports a distance-based accuracy of 0.9016 on scenarios 32–34.
These two predictors are the strongest competitors because their numbers are already near ceiling on clean line-of-sight data. A threshold rule is also the simplest possible answer to "when to sweep." We train the predictor in two versions: on labels available only from the partial logs, which is the fair comparison, and on full labels, which is an upper reference. The second version uses information a deployed system would not have.
Other learned baselines. We include three:
- Offline quantile-regression RL without conservatism, the offline counterpart of the reflector paper's method.
- A Decision Transformer trained on the same logs, following the IRS paper.
- Scalar CQL with an outage penalty added to the reward.
Ablations. Two ablations isolate the key ingredient: CODAC optimizing the mean instead of CVaR, and CODAC with $\alpha = 0$. All methods share the same logs, the same state features, the same encoder where one is used, and an equal hyperparameter search budget, with selection performed on logged data only.
What counts as beating the competitor. Beating the supervised-plus-threshold baseline means two things on the shifted test splits. The CVaR at 10% of effective rate must be higher, and outage must be lower, by margins outside seed-to-seed variation. At the same time, mean effective throughput must stay within a few percent.
No reported result was found on Lacuna for offline beam control outage or CVaR on DeepSense 6G. These headline numbers are therefore new measurements, not comparisons to published figures.
Why not just…?
Why not just use the multimodal predictor and sweep when it is unsure? This is the real threat, and it is the baseline we tune hardest. Its weakness is that softmax confidence is learned on line-of-sight data. Under night imagery or an unseen blocker, the predictor can be confidently wrong, which the authors' own non-line-of-sight caveat suggests. A threshold also cannot learn that a sweep now saves sweeps later. The shifted-split comparison settles this question directly. If the threshold rule ties on tail metrics, we report that conservative sequential control adds nothing here.
Why not just add an outage penalty to a scalar conservative critic? Reward shaping can partly imitate risk aversion. However, a scalar critic averages a two-mode outcome (clear path versus a 30 dB drop) into a single number, so it cannot tell a safe mediocre beam from a gamble with the same mean. CODAC outperformed CQL on D4RL walker2d-medium-expert (111.3 versus 98.7), but that is robotics evidence. The penalized-CQL ablation tests whether the advantage carries over to wireless.
Why not just learn the blockage dynamics and plan? The Surprising Difficulty of Search shows that search exploits errors in a learned model. Blockage events are exactly the rare states where such a model is least accurate. We therefore do not build this variant, and we flag it as a direction that stays open only if the model-free method fails.
Why not just retrain online or use a return-conditioned transformer? Live exploration drops links, which is the reason the cell-free ISAC work moved offline. The Decision Transformer is included as a baseline. If prompted with returns the logs never achieved, it should produce poor actions in blocked states, and we test this by sweeping the target return.
Decisive Checks
Training and test splits. We train on daytime vehicle-to-infrastructure logs. We test on four shifts: night scenarios, vehicle-to-vehicle scenarios, injected blockage at increasing rates, and degraded sensors (camera blur and noise, LiDAR point dropout, GPS offset).
Metrics. We report mean effective rate, CVaR at 10% of effective rate, outage probability below a fixed SNR threshold, sweeps per second, and received-power ratio against the per-slot best beam.
Expected results. The proposal is supported if three things hold. First, the CVaR gap between the full method and the supervised-plus-threshold baseline widens as blockage rate and sensor degradation increase. Second, removing conservatism, or switching CVaR to the mean, closes that gap. Third, the policy's sweep rate rises in exactly the states where its quantile spread is wide.
Further checks. Two further checks matter. The first removes sensor embeddings and keeps only radio power history. Following the Turkcell finding, we expect little loss, and that result would make the method cheaper to deploy. The second measures the shortfall between logged-data $\alpha$ selection and oracle $\alpha$ selection. A large shortfall would mean CODAC's tuning problem, not the algorithm, is what limits deployment.
Finally, we repeat the comparison on DeepMIMO O1 with the reflector. This shows whether the advantage holds when the action space grows with the number of antenna and reflector elements, which is the main stress test from the multi-antenna side.
References
- Millimeter Wave Communications With an Intelligent Reflector: Performance Optimization and Distributional Reinforcement Learning
- Decision Transformer for IRS-Assisted Systems with Diffusion-Driven Generative Channels
- Efficient Beam Selection for ISAC in Cell-Free Massive MIMO via Digital Twin-Assisted Deep Reinforcement Learning
- Conservative Offline Distributional Reinforcement Learning
- Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning
- When Vision-Language Model (VLM) Meets Beam Prediction: A Multimodal Contrastive Learning Framework
- An Explainable Failure Prediction Framework for Neural Networks in Radio Access Networks
- The Surprising Difficulty of Search in Model-Based Reinforcement Learning