Radar-Cued High-Resolution Glimpses for Distant Small Objects

Camera-based 3D detectors for driving shrink images and then downsample them repeatedly inside the network, so a pedestrian 50 meters away ends up as roughly one cell of the final feature map. Its appearance is effectively erased before any fusion with other sensors happens. Automotive radar has the opposite profile. It still returns a reliable range, and a Doppler velocity (motion measured from the frequency shift of the echo), for targets at that distance, but it cannot tell how high a return sits. A stalled car, an overhead sign, and a bridge can all produce the same radar point. We propose using each radar return as a cue that tells the camera branch where to spend expensive full-resolution computation. The network takes a small, cheap crop at native resolution along the image segment where the radar point could lie. It then decides the point's height by asking where that object would touch the road, and fuses the two sensors axis by axis rather than with a single trust weight.

The core idea fits in two sentences. Radar tells you how far, so use it to decide where to look closely in the camera image, which recovers detail that downsampling destroys. The camera then supplies direction and height, resolved by checking contact with the visible road surface. The opportunity is that every radar-camera fusion system in the literature fuses radar into camera features that have already been downsampled. Every sparse high-resolution detector, meanwhile, chooses where to look using those same degraded features.

Core Claim

Our claim is that radar returns are better cues for selective high-resolution processing than low-resolution visual heatmaps, specifically for small objects beyond about 40 meters. We expect the benefit to hold only when two other pieces are present. The first is a ground-contact test that discards cues coming from overhead structures. The second is per-axis fusion that takes range from radar and angle and height from the camera. Without the first, cues flood the detector with false positives from bridges and gantries. Without the second, the recovered image detail is diluted by a scalar blend that treats radar range and radar height as equally reliable.

Where This Sits in Prior Work

The method's starting point is QueryDet (QueryDet). QueryDet predicts coarse small-object heatmaps on low-resolution pyramid levels and runs sparse convolutions only at those locations on the high-resolution level. It reports a 3.0× speedup and +2.0 small-object AP on COCO. Its own limitation is what motivates us: if the faint target never activates the coarse heatmap, it is never revisited. We keep the sparse high-resolution mechanism and swap visual cues for radar cues, which do not degrade with apparent size. Learnable pooling (LIP) offers a different fix: it weights locations during downsampling and raises Faster R-CNN small-object AP from 21.9% to 24.0%. It still processes the whole image at reduced resolution, so it is a competitor rather than a component.

Two radar-camera systems already try to recover image detail and are the closest work. CRAFT (CRAFT) enriches radar points with image features using deformable attention over patches whose size adapts to distance. RaCFormer (RaCFormer) lets object queries sample both bird's-eye-view features and the undistorted perspective image, and it projects radar into the image as a depth hint. Both sample from backbone features computed at the network's standard reduced resolution. Neither ever revisits native-resolution pixels, so neither can recover detail that the backbone erased. RaCFormer's authors also note that missing radar height still hurts small objects, and that is the second gap we target.

On elevation, CenterFusion (CenterFusion) expands radar points into vertical pillars and associates them through a frustum around camera detections. CramNet (CramNet) samples along camera rays and uses radar as keys. Both resolve height using generic appearance along a vertical line, and the file notes that such pillars produce false associations when several objects share a range-azimuth corridor. Work on joint semantic-geometric scene parsing constrains our design. The layered street-scene model (Layered Interpretation of Street View Images) shows that enforcing ground-object-building-sky ordering along image columns cuts classification error by over 20% compared with appearance features alone. It also breaks on multi-level structures. We borrow only its weakest, most robust assumption: an object standing on the road meets the road at an image row predictable from its range. We test that assumption explicitly instead of assuming a full layered scene.

On confidence weighting, RADIANT (RADIANT) learns a scalar α to blend radar and camera depth. SAMFusion (SAMFusion) learns spatial masks over distance and visibility. CalibNet (CalibNet) scores depth reliability by RGB-depth feature similarity, and its authors report that this scoring suppresses very small instances because their depth looks noisy. That negative result directly constrains our design. A per-point quality gate applied before cueing would discard exactly the one-to-three-point returns from distant pedestrians. We therefore never gate cues by radar quality. Every return spawns a cheap glimpse, and trust is assigned afterwards, per axis.

Research Program

The system attaches to an existing radar-camera detector, with RaCFormer as the host because it is the strongest in the file. It adds three pieces. The first is a cue generator. Each radar return at range $r$ and azimuth $\phi$ is lifted to a vertical pillar spanning plausible object heights and projected into the camera. At long range this pillar covers only a short image segment, so a fixed-size native-resolution crop around it is small. The total cost scales with the number of distant returns, not with image area. Nearby returns are skipped, because large objects survive downsampling and the standard pipeline handles them.

The second piece is a glimpse encoder. A small convolutional network processes each crop at full resolution and outputs three things: an objectness score, a refined image column (azimuth), and a predicted ground-contact row. The ground-contact row comes from a lightweight road-segmentation head trained alongside detection. It is checked against the row where flat road at range $r$ should appear given camera height and pitch, and the network learns a local pitch correction to handle slopes. A cue whose visible object bottom sits well above the predicted road row is labeled as overhead structure and is not discarded outright. This is the geometric-decoupling ingredient. Semantics (where the road is) supplies the height constraint that radar cannot.

The third piece is per-axis fusion. In radar's native polar coordinates we fuse each axis with its own learned precision:

$$\hat{x}_k = \frac{w^{rad}_k x^{rad}_k + w^{cam}_k x^{cam}_k}{w^{rad}_k + w^{cam}_k}, \quad k \in \{r, \phi, h\},$$

Here $x_k$ is each sensor's estimate along axis $k$ (range, azimuth, height), and $w_k$ is a predicted inverse variance. Radar contributes no height estimate, so $w^{rad}_h$ is fixed to zero, and the camera's range weight is expected to fall with distance. The fused glimpse becomes an extra object query for the host decoder. Because the glimpses enter as additional queries, the host model is otherwise unchanged, which keeps the comparison clean.

Cue recall is limited by radar itself. CRAFT reports much smaller gains on non-metallic, low-reflectivity objects than on cars, so radar may simply miss some pedestrians. We therefore also run a union variant that takes radar cues plus QueryDet-style visual cues, and we report which cue source produced each recovered detection.

Baselines

The naive reference point is the unmodified host, RaCFormer, at its standard input resolution. RaCFormer reports 64.9% mAP and 70.2% NDS on the nuScenes test set. We retrain it ourselves on the validation protocol so that all variants share data, schedule, and backbone. For a lighter host, we also add glimpses to RCTrans, which reports 59.4% NDS and 52.0% mAP on nuScenes validation with a Swin-T backbone (RCTrans).

The strongest compute-matched competitor is the same host run at higher input resolution, with input size chosen so frame rate matches our method on the same GPU. A second high-resolution competitor replaces the host's downsampling layers with learnable importance pooling. This mirrors the LIP result, which lifted small-object AP on COCO from 21.9% to 24.0% without enlarging feature maps.

The strongest competitor on the cueing question is QueryDet-style visual cueing grafted onto the host. It uses the identical glimpse encoder and the same crop budget, but takes cues from coarse visual heatmaps instead of radar. If radar cues do not beat this baseline on objects beyond 40 meters, the central claim fails.

Ablations isolate each ingredient. One spends the same number of crops at random locations along the road band. One replaces per-axis fusion with RADIANT's scalar α. One removes the ground-contact test and uses plain CenterFusion-style pillars. One adds a CalibNet-style radar quality gate before cueing, to test whether gating suppresses distant small objects as CalibNet's authors observed in their setting. Every variant gets the same training schedule and the same hyperparameter search budget on a held-out split. Beating a baseline means a gain in distance-binned AP beyond 40 meters for pedestrian, bicycle, and traffic-cone classes that exceeds three-seed variation, at no worse frame rate. Overall NDS must not drop. No reported distance-binned numbers for these hosts were found on Lacuna, so all such bins will be measured fresh.

Why not just…?

Why not just feed the camera branch higher-resolution images? This is the most serious threat. If raising resolution at matched frame rate recovers the same distant-object AP, the cueing machinery is unnecessary. The evidence against it is cost. QueryDet reports that adding a dense high-resolution pyramid level drops throughput from 13.6 to 4.85 FPS, and radar-camera hosts already run at about 12–16 FPS. The compute-matched high-resolution baseline settles the question directly, and we will report the full accuracy-versus-FPS curve rather than a single point.

Why not just use QueryDet's visual heatmaps as cues? QueryDet's own limitation is that a target missed on the coarse map is never refined, and distant objects are exactly the ones whose coarse activation is weakest. The risk on our side is that radar misses low-reflectivity pedestrians. The matched visual-cue baseline and the union variant measure both effects, and the recall breakdown by cue source shows which sensor finds which objects.

Why not just rely on RaCFormer's perspective-view sampling with radar depth hints? RaCFormer already returns to the perspective plane, but it samples backbone features at reduced resolution, so an object that occupies one feature cell gets no extra detail. Its authors also flag radar height as an unsolved problem for small objects. Our method is added on top of RaCFormer, so any gain is by construction beyond what its sampling recovers.

Why not just learn a scalar trust weight, as RADIANT does? A scalar cannot express that radar is precise in range and blind in height while the camera is the reverse. The scalar-α ablation measures whether this matters, mainly through translation error (mATE) and height error on distant objects. If per-axis fusion gives no improvement over the scalar, we will report that and drop the added complexity.

Proposed Experiments

The main experiment is on nuScenes, whose conventional radars measure no elevation. We report mAP, NDS, mATE, and mAVE overall, plus distance-binned AP in 10-meter bins out to the annotation limit, along with FPS on a single RTX 3090 for direct comparison with the throughput figures reported for RaCFormer and RCTrans.

The decisive check for the elevation component uses View-of-Delft. Its 4D imaging radar measures elevation natively, which gives us ground truth for height. We discard the measured elevation at input, run the ground-contact test, and score how well it separates overhead returns from road-level returns, using the hidden elevation as the label. We also measure how many false detections each variant creates from overhead returns. If the ground-contact test cannot recover elevation better than plain pillars on this data, the geometric ingredient is not doing its job.

A stress test on sloped or multi-level road segments probes the flat-road assumption that the layered-scene work identifies as fragile. On View-of-Delft we can measure these segments directly using its LiDAR. On nuScenes we use scenes tagged by elevation change.

The outcome that would most clearly validate the proposal combines three results. Radar-cued glimpses should beat both compute-matched high resolution and matched visual cueing on pedestrians and cyclists beyond 40 meters. Removing the ground-contact test should raise false positives on View-of-Delft overhead returns. Adding a pre-cue radar quality gate should lower distant small-object recall.