Radar-Led, Contrast-Gated Attention for Ultra-Low-Resolution Thermal Sensing

Very coarse thermal sensors, with grids as small as 8×8 or 32×24 pixels, are attractive for monitoring people in homes, hospitals, and offices. They protect privacy at the hardware level, because a person shows up only as a warm blob with no face or texture. That same coarseness creates two weaknesses. First, the sensor cannot say precisely where a person is. Second, when the room is close to skin temperature, for example near a radiator or a sunlit window, the blob fades into a flat field and the sensor cannot say whether a person is there at all. A second privacy-preserving sensor, millimetre-wave (mmWave) radar, has the opposite profile. It reports precise 3D points but produces "ghost" reflections that look like people. We propose to fuse the two with cross-modal attention, a mechanism in which features from one sensor ask questions of the other's features. Each radar detection asks the coarse thermal grid, "is there body heat where I think a person is?" How much the answer counts is set by a gate driven by the measured local thermal contrast, so the thermal evidence is switched off when the physics says it is uninformative.

The question behind this design has not been tested. Should the modality that asks the questions be the one that carries spatial precision, and does that choice flip as thermal resolution drops? We expect two things. Radar-led querying should overtake thermal-led and symmetric attention once thermal resolution falls to array scale. A gate grounded in physical contrast should hold up better on unseen warm scenes than a gate learned only from features.

Core Claim

Our claim is that at ultra-low thermal resolution the best fusion is asymmetric. The query comes from radar, the modality that can localize. Thermal acts only as a gated confirmation signal. The gate takes as input a cheap physical contrast statistic rather than relying only on learned features.

Prior work splits along the two halves of this claim. Robust Human Detection under Visual Degradation via Thermal and mmWave Radar Fusion (UTM) fuses thermal images and radar depth images with uncertainty weighting from Monte Carlo dropout and reports 0.644 mAP50:95. It uses standard-resolution thermal cameras and no query-key attention. It therefore motivates the problem, since it documents thermal contrast collapse, but it leaves the low-resolution regime untested. Two other papers supply the attention mechanisms we reuse. Object Tracking by Jointly Exploiting Frame and Event Domain uses bidirectional cross-attention between frames and event-camera streams. Hierarchical Multimodal Transformer to Summarize Videos uses one-directional attention in which vision queries audio. Neither paper asks which direction the queries should run when one sensor loses its spatial detail. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention introduces a learnable gate that stops one stream from washing out the other. We reuse that gate but feed it an externally measured, sensor-physics quantity.

Two thermal-only papers constrain the design. Privacy-Preserving Person Detection Using Low-Resolution Infrared Cameras shows that class activation maps cannot draw boundaries at 32×24. This is a warning that spatial attention maps over coarse thermal tokens may not localize anything, and it is the reason localization is assigned to radar. UPPET shows that ViTPose-h drops from 92.0 AP to 57.8 AP across thermal sensors, with resolution mismatch as a main cause. This motivates a gate input that means the same thing on every sensor.

Research Program

The model encodes the thermal stream as a small grid of tokens. Each token combines the raw temperature, the temporal-difference image $|I[n]-I[n-1]|$ (which raised 8×8 thermopile accuracy from 83.3% to 99.2% in Convulsive Movement Detection using Low-Resolution Thermopile Sensor Array), and a reconstruction residual from an auto-encoder trained on empty rooms, following the unsupervised background model of Privacy-Preserving Person Detection Using Low-Resolution Infrared Cameras. Radar points are clustered into candidate tokens. Each candidate carries its 3D position, Doppler velocity, and point statistics.

Each radar token $r_i$ forms a query and attends over the thermal tokens, restricted to those inside its projected footprint. The result is a thermal context vector $t_i$. The radar token is then updated as

$$\tilde r_i = g_i\, t_i + (1-g_i)\, r_i, \qquad g_i = \sigma\!\left(w^\top r_i + \alpha\, c_i + \beta\right),$$

where $g_i \in [0,1]$ is the gate and $\sigma$ is the logistic function. The gate's inputs are the radar feature $r_i$ itself, through learned weights $w$, and a contrast statistic $c_i$, through learned scalars $\alpha$ and $\beta$. The contrast $c_i$ is the mean temperature in the radar token's footprint minus the scene median, divided by a per-sensor noise estimate taken from empty-room frames. Normalizing by sensor noise is what lets $c_i$ carry the same meaning across sensors. A small head then scores each updated radar token and regresses its box. The box is expressed in the original image coordinates, so 8×8 thermal input still yields evaluable boxes. We target a parameter budget comparable to the 32,642-parameter attention model of BiCrossNet, so the head can run on embedded hardware.

The main data source is the UTM dataset, which has 24,241 frames across 10 indoor scenes with darkness, smoke, and glare. We produce low-resolution thermal inputs by area-averaging to 32×24 and 8×8, then adding noise and blur matched to the empty-room statistics of real 32×24 arrays in Distech-IR and FIR-Image-Action. Because synthetic downsampling is not a real thermopile, we will also co-mount a 32×24 array and an 8×8 Grid-EYE beside a radar for a small real capture. That capture serves as a check that the simulated results transfer. All results are reported per contrast bin, with frames grouped by their measured contrast, so that warm-scene behaviour is visible rather than averaged away.

Baselines

The reference points with no fusion are a radar-only detector and a thermal-only YOLOv5s, each trained at every resolution. The radar-only detector is the decisive one at 8×8. If fusion cannot beat radar alone there, then the thermal array adds cost but no information, and we would report that.

The strongest existing method is UTM (Robust Human Detection under Visual Degradation via Thermal and mmWave Radar Fusion). It reports 0.644 mAP50:95 at native resolution, and we will first reproduce that number. No reported result found on Lacuna for UTM or any thermal-radar fusion at 32×24 or 8×8, so we retrain UTM at each resolution ourselves. To keep the comparison fair, UTM receives the same thermal input channels as our model, including the temporal difference and auto-encoder residual, along with the same radar projection, the same held-out-scene splits, the same epoch count, and the same hyperparameter search budget of an equal number of trials per method. Our model beats UTM if, at 32×24 and 8×8, it exceeds retrained UTM on mAP50:95 by more than the seed-to-seed standard deviation, both overall and in the lowest-contrast bin, while staying within noise of UTM at native resolution.

Three variants isolate the attention design. The first is plain early concatenation, which stacks the upsampled thermal image as an extra channel of the radar depth image fed to YOLOv5s. The second is ungated bidirectional cross-attention in the style of Object Tracking by Jointly Exploiting Frame and Event Domain. The third is thermal-led one-directional attention in the style of Hierarchical Multimodal Transformer to Summarize Videos.

Two gate ablations isolate the key ingredient. One sets $\alpha=0$, which leaves the purely learned gate of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention. The other sets $w=0$, which leaves a purely physical gate. All variants share the same thermal and radar encoders and the same detection head, so they differ only in how the two streams are combined.

Why not just…?

Why not just concatenate the thermal image as an extra input channel? Audio-Visual Classification of Sports Types reached 96.11% by concatenating thermal and audio features into a kNN classifier. Simple fusion is clearly a real threat. The counter-evidence comes from Object Tracking by Jointly Exploiting Frame and Event Domain, whose ablations found naive concatenation insufficient. That evidence is for event cameras, though, not coarse thermal. Our concatenation baseline settles the question directly. If it matches our model in every contrast bin at 8×8, then the attention machinery is unjustified and we will say so.

Why not just use radar alone once thermal is this coarse? At 8×8, a single thermal pixel can cover most of a person, so this objection is plausible. Our hypothesis is that thermal is still useful as presence confirmation, specifically for rejecting radar multipath ghosts, which UTM identifies as a radar failure mode. The test is the false-positive rate on radar candidates with no matching ground-truth person. Fusion should reduce it relative to radar-only in high-contrast bins and should match radar-only in flat bins, where the gate ought to close.

Why not just let a learned gate, or modality dropout, figure out when to trust thermal? In the training scenes this probably works equally well. The case for the contrast input concerns new scenes and new sensors, where learned features shift, as UPPET's drop from 92.0 to 57.8 AP illustrates. The $\alpha=0$ ablation, evaluated on held-out warm scenes and on the real-sensor capture, decides this. If the learned gate matches ours there, we drop the contrast term.

Why not just apply temporal differencing with a small CNN? That approach reached 99.2% on convulsion detection, but it worked on isolated, vigorous motion in an otherwise empty bed. It gives no signal for a person sitting still. We include temporal differencing as an input to every method, so any gain we report comes on top of it rather than replacing it.

Proposed Experiments

The first decisive check is the resolution sweep, run at native resolution, 32×24, and 8×8, comparing thermal-led, bidirectional, and radar-led attention. Our claim predicts a crossover. Thermal-led or bidirectional attention should be competitive at native resolution, and radar-led attention should win at 8×8. If no crossover appears, the central hypothesis is wrong.

The second check stratifies results by contrast. We bin test frames by $c_i$ and also synthesize thermal inversion by shrinking the dynamic range and raising background temperature. This should show three things: the gate closing as contrast falls, the ungated model's mAP collapsing below radar-only in flat bins, and the gated model staying at or above radar-only.

The third check tests transfer. Models trained on simulated low-resolution UTM data are evaluated on the real co-mounted array capture, and we report the drop for each fusion variant. This extends UPPET's cross-sensor protocol to fusion.

Finally, we report parameters, latency, and RAM on an embedded CPU. These are compared against UTM's 5–7 FPS on Jetson and against BiCrossNet's 9.16 ms footprint as an edge reference. A model that wins on accuracy but cannot run on a microcontroller-class device fails the deployment goal that motivates low-resolution sensing in the first place.