Automotive radar is the only common driving sensor that still sees well at night and in rain. Yet nearly every learning pipeline receives it as a few dozen points, after the sensor's firmware has run a threshold detector and thrown away the phase of the returned wave. Methods that add synthetic "virtual" 3D points to fill in sparse sensors start from these impoverished points. So do methods that upsample radar. Both end up guessing geometry from leftovers. This proposal goes back one step in the radar signal chain to the raw sampled waveform, which is complex-valued: each sample has an amplitude and a phase. A radar's range–azimuth image is computed from these samples by a Fourier transform. Multiplying the raw samples element by element with a learned complex mask is therefore exactly equivalent to convolving the whole range–azimuth image with a kernel as large as the image itself.
That equivalence links three problems that are usually treated separately. Convolutional networks see only a local window, and global attention costs grows with the square of the input size. Complex-valued networks preserve wave physics but pay roughly four real multiplications for every complex one. Virtual points generated from thresholded radar inherit its ghosts and gaps. A learned complex gate placed before the FFT the radar already computes gives a global receptive field over range, angle and Doppler. It costs one complex multiply per sample per feature channel. We then emit virtual 3D points directly from the gated, phase-aware representation, using the camera only to supply the vertical extent that a 2D radar cannot measure.
Core Claim
A small bank of learned, content-modulated complex gates applied to the raw radar sample cube (antenna × chirp × fast-time sample) should produce better range–azimuth features than local convolutions or attention at equal compute. Points sampled from a learned occupancy over those features, rather than from fixed-threshold detections, should give a standard 3D point detector more accurate and less ghost-ridden input. The mechanism rests on one identity. Let $x$ be the raw sample cube, $\mathcal{F}$ the multi-axis FFT the radar already performs, $g$ a learned complex gate of the same shape, and $\circledast$ circular convolution. Then
$$\mathcal{F}(g \odot x) = \tfrac{1}{N}\,\mathcal{F}(g) \circledast \mathcal{F}(x),$$
where $N$ is the number of samples. A pointwise gate in sample space is a full-image convolution in range–Doppler–azimuth space. Its cost is $4CN$ real multiplications for $C$ gate channels. A $k \times k$ complex convolution costs $4k^2CN$, and dense attention costs order $N^2$.
Classical radar processing already applies a fixed, real version of this, the tapering window that suppresses sidelobes. We make the window complex, multi-channel, separable by axis (as ABFNet separates its frequency filters by direction), and lightly conditioned on scene context through a low-rank modulation computed from a coarse summary of the cube.
Relation to Prior Work
The virtual point literature motivates the target and supplies the lifting recipe. Multimodal Virtual Point 3D Detection (MVP) lifts camera mask pixels to 3D by borrowing depth from the nearest LiDAR return. It reports 66.4 mAP and 70.5 NDS on the nuScenes test set, with the largest gains on small and distant objects. However, its depth rule assumes fronto-parallel surfaces, and it needs LiDAR at test time. We invert MVP's division of labour. Radar supplies range and azimuth, which it measures accurately. The camera mask supplies only height extent, which radar lacks.
CenterFusion expands each firmware radar point into a fixed vertical pillar. GET-UP and mmDEAR upsample post-threshold radar points with learned point decoders. All three start after the threshold detector. The file notes GET-UP's sensitivity to the number of generated points and mmDEAR's failure under multipath ghosts. Both look like symptoms of starting from points that are already wrong. Our points are emitted before any hard threshold, from a representation that still carries phase.
Vision meets mmWave Radar (CRUW3D) argues that raw complex radar tensors separate targets such as pedestrians and metal poles that point clouds confuse. It also releases raw sample data, which this proposal depends on. It does not generate 3D points for standard detectors. Its radar baseline, RODNet, works on 2D range–azimuth maps.
The receptive-field literature supplies the method template and its constraints. Butterworth as Attention (ABFNet) obtains a global receptive field by learning anisotropic gates on FFTs of real images. It reports being 2.4× faster with 2.3× less peak memory than ARConv. It also observes that ordinary spatial operators misbehave in the frequency domain and that Vision Transformers blur edges. We reuse its gating idea. The difference is that our "frequency domain" is the sensor's native measurement space, so the gate acts on physical, phase-bearing data rather than on transformed real features. CCNet and Compact Generalized Non-local Network represent the attention-based alternatives whose cost we aim to avoid.
The complex-network literature constrains the design. Work summarised in the file (Zhang et al., Wu et al., Nazarov and Burnaev) documents that complex layers inflate multiply counts and that pruning does not recover the latency. We therefore confine complex arithmetic to one elementwise gating stage plus FFTs the radar already computes. Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks shows that wide complex networks with split activations collapse to ordinary real networks. Our complex stage is linear (a gate followed by an FFT) with no complex activation at all, so that failure mode does not arise inside it.
Shift-Equivariant Complex-Valued Convolutional Neural Networks reports that mapping complex features to real ones is a bottleneck: MLP projections are slow and magnitude projections discard phase. We use a conjugate-product projection instead. Pairs of gate channels $z_a, z_b$ are combined as $z_a \bar{z}_b$, which costs one complex multiply. It removes the arbitrary absolute phase but keeps relative phase, which encodes angle-of-arrival consistency and Doppler.
Research Program
The model has three parts. The first is the gate bank. It holds $C$ complex gates factored along the antenna, chirp and fast-time axes. Each factor is a static learned profile plus a rank-$r$ modulation predicted by a tiny real network from a pooled, low-resolution magnitude summary of the cube. The factorization keeps parameters small and lets the gates act as learned range, angle and Doppler windows.
The second part runs the radar's own FFTs on the gated cubes. It applies the conjugate-product projection and a shallow real-valued 2D network on the resulting range–azimuth–Doppler features. This network predicts a per-cell occupancy probability and a radial-velocity estimate.
The third part is the emitter. It samples virtual points from the occupancy map with a fixed point budget, so no hand-set threshold is needed. It then projects each point's azimuth into the camera image to find a column and uses an off-the-shelf 2D instance mask to assign height samples, giving each point a vertical extent. Points falling in no mask receive a learned default height distribution. Each point carries its occupancy, velocity and relative-phase features. The output feeds an unmodified point-based 3D detector.
Training supervises occupancy with LiDAR occupancy projected to range–azimuth and supervises detections with the 3D boxes. LiDAR is used only during training.
The analysis asks three questions. Does the learned gate acquire genuinely complex structure, meaning non-constant phase, or does it collapse to a real taper? Where in range does the advantage appear? Do ghost points, defined as emitted points with no LiDAR support within a tolerance, fall relative to firmware-style detections?
Baselines
The do-nothing reference is the camera-only DD3D baseline from CRUW3D, which reports 25.41% AP on cars at 0.7 IoU and 8.29% AP on pedestrians. We rerun it on our split so that the numbers are directly comparable.
The strongest radar-only competitor in the file is RODNet, for which CRUW3D reports 32.72% detection AP. That figure is a 2D range–azimuth detection score, not a 3D box AP, so it cannot be compared directly with our results. We will retrain RODNet on the same split and the same raw tensors, with the same tuning budget and GPU hours. Its outputs will be lifted with the same camera-height emitter and fed to the same 3D detector. This isolates what the gated front end contributes.
The strongest point-densification competitors are a CenterFusion-style pillar expansion of thresholded detections and a GET-UP-style learned upsampler. Both are reimplemented on CRUW3D detections obtained by running a standard threshold detector on the raw data. They feed the same detector and receive equal hyperparameter search, including GET-UP's point-count setting, which the file flags as sensitive. As an upper reference that uses privileged input, we run MVP-style lifting with test-time LiDAR. This bounds what radar-seeded points could hope to reach.
Ablations isolate the key ingredient. The first replaces the complex gate with an unconstrained real 2×2 linear map per sample on the in-phase and quadrature channels. The second moves the same gate to after the FFT, acting on magnitude maps as ABFNet does. The third removes the context modulation. The fourth swaps the conjugate-product projection for magnitude and MLP projections. The fifth replaces the gate bank with a full complex convolutional front end in the style of DCCRN, matched in multiply count.
Beating the field means two things. First, higher 3D AP than the best lifted-RODNet and densification baselines at equal or lower measured latency on the same GPU. Second, a gain concentrated, as the mechanism predicts, at long range and on pedestrian classes, where global context and phase matter most.
Why not just…?
Why not just feed in-phase and quadrature as two real channels to a standard CNN? This is the most serious alternative. A per-sample real 2×2 map is strictly more expressive than a complex gate, and the CRUW3D baselines already process raw tensors this way. Evidence from Enhancing RF Sensing with Deep Learning and the SAR results in Shift-Equivariant CVNNs suggests that coupled phase helps. However, none of that evidence holds the receptive field and compute fixed. Our first ablation does. If the real 2×2 gate placed before the FFT matches the complex gate, we will report that the benefit comes from where the gating happens, not from complex algebra, and narrow the claim accordingly. The complex version would then survive only through its smaller parameter count.
Why not just apply ABFNet-style spectral gating or a Vision Transformer to the range–azimuth image? Gating after the FFT acts on magnitudes and loses the inter-antenna phase relationships that determine angle. Attention on a full-resolution range–azimuth–Doppler volume scales quadratically, and ABFNet reports that it blurs edges. The post-FFT gate ablation and an attention baseline matched in FLOPs settle this directly on long-range AP and latency.
Why not just densify the firmware points, as GET-UP and CenterFusion do? Points that the threshold detector missed or hallucinated cannot be recovered downstream. mmDEAR's degradation under multipath ghosts is consistent with this. We measure ghost rate and recall against LiDAR for both pipelines. If densified firmware points match our recall at long range, the argument for going back to the raw samples fails.
Why not just train a full complex-valued network on the raw cube? The overhead literature in the file shows complex convolutions multiply cost without proportional benefit. Split activations also risk the collapse proved in Real-Valued Backpropagation is Unsuitable. The multiply-matched DCCRN-style ablation tests whether that extra depth buys accuracy that a single linear complex stage cannot.
Proposed Experiments
All main experiments use CRUW3D, because it is the only benchmark in the file that ships raw radar samples. nuScenes provides only post-threshold radar points, so it serves only as a secondary check of the emitter on firmware points. We train on CRUW3D's standard split and evaluate separately on its adverse-condition subset, which makes up about 30% of frames.
The metrics are per-class 3D AP binned by range, emitted-point recall and ghost rate against LiDAR occupancy, measured latency and peak memory, and the confusion rate between pedestrians and pole-like static objects.
The decisive checks are four. First, the complex gate before the FFT must beat both the real 2×2 gate and the post-FFT gate at equal compute. Second, the beyond-40 m AP gain over lifted RODNet must exceed the near-range gain. Third, the learned gates must show non-trivial phase structure; if they collapse to real tapers, the complex framing is unnecessary. Fourth, the end-to-end latency overhead over the radar's existing FFT pipeline must stay within a small constant factor. If the first check fails, the proposal reduces to the weaker but still useful claim that learned global gating should happen in the sensor's native domain.
References
- Multimodal Virtual Point 3D Detection
- CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
- GET-UP: GEomeTric-Aware Depth Estimation with Radar Points UPsampling
- mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction
- Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Compact Generalized Non-local Network
- An optical neural chip for implementing complex-valued neural network
- Rethinking Complex-Valued Deep Neural Networks for Monaural Speech Enhancement
- Bayesian Sparsification of Deep C-valued Networks
- Real-Valued Backpropagation is Unsuitable for Complex-Valued Neural Networks
- Shift-Equivariant Complex-Valued Convolutional Neural Networks
- DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
- Enhancing RF Sensing with Deep Learning: A Layered Approach