Radio-Supervised Scene Residuals for Open-Set mmWave Blockage Detection

High-frequency radio links, meaning millimeter-wave and terahertz links that travel in narrow, almost optical beams, fail abruptly when anything physical crosses the straight path between the base station and the user. Cameras mounted on base stations can see these blockers coming. Current vision-aided systems, however, only see the objects they were trained to name: cars, trees, poles. Our proposal swaps the question "what object is this?" for "what in this view is not part of the empty, static scene, and does it sit inside the radio path?" We learn what the empty scene looks like, flag whatever the model cannot reconstruct, and lift those flagged pixels into metric 3D using the known positions of the base station and the user. The step that sets this apart from existing work is that the radio link supervises the anomaly detector. Every measured drop in received power is a free label saying "something in this cone mattered." That label teaches the detector which visual deviations block the signal and which ones do not, such as shadows, glare, rain, or swaying foliage.

Two fields gain from this. Wireless beam management gets a blockage detector that does not depend on object classes. Anomaly detection gets a fix for a known weak point: unsupervised anomaly detectors have no negative examples with which to set their decision threshold. Here the physical channel supplies those examples continuously and at no cost.

Core Claim

A fixed base-station camera paired with GPS on the user gives a cheap, exact geometric gate. The base station's line of sight to the user projects into the camera image as a small cone around the user's pixel position. Its angular width is set by the first Fresnel zone, the ellipsoid around the direct ray that must stay clear for the link to hold. That width is widest close to the antenna and narrows with distance, roughly as the square root of wavelength over range.

We claim that a reconstruction-based detector, trained on views of the static scene and gated by this cone, predicts link blockage better than closed-set detection on blockers outside the known classes. We further claim that fine-tuning its pixel-level decision stage on labels derived from received power outperforms the same detector with a fixed or synthetic-anomaly threshold. The second claim is the part that is new. The first is the part a skeptic will attack with classical background subtraction, and the program is built so that this attack is settled empirically rather than argued.

What Exists and Where This Sits

Three lines of work use vision for beam management, and all of them couple vision to the radio through closed-set recognition.

Vision-Assisted Digital Twin Creation for mmWave Beam Management builds a 3D point-reflector scene. It uses monocular depth, Segment Anything masks, and YOLO class labels, runs ray tracing over that scene, and corrects the result with a learned linear layer. If an object is not detected, it is missing from the twin, and the ray tracer reports an open path.

Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning fuses GPS, image, and LiDAR features directly into a beam classifier. The authors state that their evaluation focuses on line-of-sight conditions.

When Vision-Language Model (VLM) Meets Beam Prediction aligns image and LiDAR embeddings contrastively and gates between modalities depending on day or night conditions.

These papers motivate the proposal: they show that vision substantially reduces beam-search cost. They also constrain it: the 0.5 to 1.0 ms inference latency reported by Mollah et al. rules out iterative diffusion reconstruction in the loop. Our proposal keeps their beam predictors intact and adds a class-free blockage signal alongside them.

On the trajectory side, Dynamic Blockage Pre-Avoidance using Reconfigurable Intelligent Surfaces pre-selects a reflecting surface by extrapolating blocker trajectories. It assumes perfect bounding boxes. Our detector is intended to be the perception front end that this work abstracts away.

The anomaly side supplies the method. DRÆM pairs a network that reconstructs a clean image with a second, discriminative network that learns pixel-level differences. It trains on synthetic pasted-texture anomalies and reports 98.0 image AUROC and 68.4 pixel AP on MVTec AD. We reuse its two-network structure because both passes are single forward passes, which keeps latency compatible with camera frame rates.

SplatPose & Detect is the closest precedent for joining coordinate registration with reconstruction residuals. It registers a query image to a 3D Gaussian model of a normal object before computing residuals. It works on tabletop objects, needs about 5 seconds per view, and has no notion of a radio path.

Two known failure modes of reconstruction detectors explain why radio labels help. The first is the thresholding trap identified alongside MVTec AD: without anomalous validation data, there is no principled way to set a cutoff. The second is the trade-off between faithfully copying anomalies and blurring legitimate high-frequency texture (the "identical shortcut" discussed by Fucka et al.). Outdoors, the second mode would fire on foliage and asphalt. Radio labels attack both problems at once.

Section 2 of the research file found no paper joining reconstruction-based anomaly detection with high-frequency blockage. The specific mechanism of using the channel to supervise the anomaly detector's decision stage also does not appear in any cited work.

Research Program

The system has three components.

Geometry. The first component maps the user's GPS position into camera pixels. In the DeepSense 6G vehicle-to-infrastructure scenarios the camera sits at the base station, so the base-station-to-user ray degenerates to a point in the image, the user's pixel location, surrounded by the Fresnel cone. We estimate the camera's pose in the GPS frame by solving a perspective-n-point problem. The correspondences are pairs of user GPS fixes and the pixel centroid of the user vehicle, which is detected only for calibration. This is the same "solve the fixed transform from paired observations" logic as hand-eye calibration in Han et al.. Calibration is refit periodically so that it absorbs drift. Monocular depth gives each residual pixel a range, and only residuals nearer than the user count as blockers. The user vehicle itself is excluded by its projected footprint.

Reconstruction. The second component is a per-site reconstruction network. It is trained to map any frame to the static empty scene, using temporal-median background targets and synthetic pasted-object corruptions in the DRÆM style.

Radio supervision. The third component fine-tunes the discriminative stage on radio labels. A frame is labeled "blocked" when the best-beam received power falls more than a margin below the power expected from distance and from the user's recent power trend. It is labeled "clear" otherwise. The loss is applied only to the residual mass inside the Fresnel cone, a multiple-instance setup: the label says something in the cone mattered, not which pixel. As a result, the network learns to discount residual types that never coincide with power drops.

The analyses we will run follow from this design. We will measure how often residual mass in the cone predicts power drops, and how that relationship differs between day and night. We will examine which residual categories the radio fine-tuning learns to suppress. We will measure how prediction lead time trades against accuracy when the cone is advanced along the user's GPS trajectory.

Baselines

The naive reference point is a radio-only predictor. It declares blockage whenever the previous received-power sample dropped by the same margin used to create labels. This tells us whether vision adds anything beyond the link's own short memory. A second naive point always predicts line of sight.

The strongest existing method is the closed-set digital twin of Vision-Assisted Digital Twin Creation. We run its YOLO, SAM, and depth pipeline and call a frame blocked when any detected object's reconstructed points fall inside our Fresnel cone. That reuses its perception and gives it our exact geometry, so the only difference is closed-set versus residual-based foreground. Its reported downstream figure is a Distance-Based Accuracy (DBA) of 0.92 on seen Scenarios 32 to 34 and 0.93 on unseen Scenario 31, against 0.80 for an end-to-end network on the unseen site. No reported result was found on Lacuna for blockage detection itself on DeepSense 6G. We will therefore report blockage AUROC for this baseline ourselves, and also DBA when its beam predictions are replaced by a fallback beam on frames our detector flags.

The classical competitor is adaptive mixture-of-Gaussians background subtraction plus the same geometry and depth gate. Its threshold is tuned on the same radio-labeled validation split. This is the comparison that tells us whether learning is needed at all.

A class-agnostic segmentation baseline uses Segment Anything masks that do not match the static background, again with the same gate.

An end-to-end supervised baseline trains the image-plus-GPS architecture of Mollah et al. to output the blockage label directly. That paper reports 98.19% top-13 beam accuracy but no blockage figure.

The ablations isolate each ingredient in turn. We remove radio fine-tuning and use DRÆM with a fixed percentile threshold. We remove the geometric gate and use residual mass over the whole frame. We remove the depth ordering.

All learned methods get the same site data, the same training and validation splits, and the same hyperparameter search budget. Beating a baseline means a higher blockage AUROC with non-overlapping bootstrap confidence intervals, on the full test set and on the subset of events whose blocker falls outside YOLO classes.

Why not just…?

Why not just use background subtraction against a static reference? This is the real threat. With a fixed camera, it may produce foreground masks nearly as good as a learned reconstructor in daylight. We expect it to fail where the scene changes without anything blocking the signal: headlights and shadows at night (Scenarios 33 and 34), and moving vegetation. Those failures should show up as false alarms that radio labels can expose. If background subtraction with our geometry and a radio-tuned threshold matches the full system, we will report that. The contribution then narrows to the geometry and the radio-calibration recipe, and the generative component is dropped. The ablation that applies radio threshold-tuning to background subtraction is included specifically to settle this.

Why not just use an open-vocabulary or class-agnostic detector? Segment Anything removes the class-name bottleneck but not the second problem: deciding which segments matter to the channel. A mask on a shadow or a reflection still looks like an object. We test this directly with the class-agnostic baseline. The prediction is that it matches us on recall but loses on precision in the night scenarios.

Why not just train an end-to-end blockage classifier on image and GPS? It would use the same radio labels we use. The digital-twin paper shows that such networks drop from 0.87 to 0.80 DBA on an unseen site, consistent with memorizing site-specific appearance. We repeat that cross-site test for blockage. If the end-to-end classifier generalizes as well as our system, the explicit geometry is not earning its place.

Proposed Experiments and Decisive Checks

The data are DeepSense 6G Scenarios 31 to 34, with Scenario 31 held out as an unseen site. The first decisive check comes before any modeling. Mollah et al. note that these scenarios are largely line of sight, so we will count power-drop events under our labeling rule and hand-audit a sample against video. If fewer than a few hundred genuine blockage events exist, the main claim cannot be tested on this data. In that case we would narrow the study to the cross-site and night subsets that do contain events, and state that limitation explicitly.

We will also hand-label each event's blocker as YOLO-class or out-of-class. As a controlled open-set test, we will additionally rerun every closed-set baseline with trucks and buses deleted from the detector's vocabulary.

The metrics are:

  • blockage AUROC and precision at fixed recall, split by day, night, and out-of-class blockers;
  • AUROC as a function of prediction lead time;
  • downstream DBA when flagged frames trigger a fallback beam;
  • per-frame latency, which must stay within the camera frame interval.

The proposal succeeds if three things hold. Radio fine-tuning beats the fixed-threshold ablation. The full system beats the closed-set digital-twin gate on out-of-class events. And it beats radio-tuned background subtraction at night. If only the first holds, radio supervision is still a useful calibration method, but the case for generative reconstruction fails, and we will say so.