Telemetry-Guided Foveated Thermal Detection for Retrofitted Vehicles

Cheap thermal cameras paired with small edge computers can now give old trucks and snowplows fog-proof obstacle detection. In the reference system, a FLIR camera feeds a pruned detector on an NVIDIA Jetson board and scores F1 0.8958 in foggy night conditions, where an RGB camera scores 0.4737 (Low-Cost Infrared Vision Systems). That system treats every pixel of every frame the same way. To fit the board, it shrinks the network from 37M to 2.5M parameters and runs it on a downsampled full frame. The objects that matter most at highway speed are distant ones, a pedestrian or stalled car a few hundred meters out, and in thermal imagery these are tiny. More than 81% of targets in the RGBT-Tiny benchmark are smaller than 16×16 pixels (Visible-Thermal Tiny Object Detection), and uniform downsampling is exactly what erases them.

We propose to spend a fixed per-frame compute budget unevenly, the way attention does in the brain. Every frame gets a cheap low-resolution "peripheral" pass. A few high-resolution "foveal" tiles are chosen before the frame arrives, using the vehicle's own state: its speed, its heading, and what was seen in the last frame. At 90 km/h the fovea sits on the far road band near the horizon. At 20 km/h it widens to the near field. The claim is simple. On the same cheap board and at the same measured latency, context-chosen high-resolution tiles detect distant hazards earlier than uniform processing, bottom-up saliency, or plain tracking. The same selection signal can also decide which detections get shown to the driver.

Core Claim

The research file makes the gap concrete from two sides. The retrofit side (Low-Cost Infrared Vision Systems, Smart Rail Guard, IOT – Based Industrial Safety and Protection System) runs uniform loops: every frame or sensor sample gets full, identical processing regardless of context. The attention side has found three relevant things. Selection in the brain happens early, inside primary sensory cortex. It is driven top-down by task context. And a neuron's firing before the stimulus predicts how strongly it will gate (Early selection of task-relevant features through population gating). Attention also shifts early visual tuning toward the task-relevant spatial scale (Attention Alters Population Spatial Frequency Tuning). We use these findings as design motivation, not as models to replicate. Three design choices follow. The gate is computed from context before the frame, so that skipping can be scheduled in coarse tiles that a commodity GPU actually runs faster on. The gate chooses resolution per region, not just on or off. And the gate's input is task state, meaning vehicle telemetry, rather than the image alone.

Stated formally, we split each frame into tiles $i$ and let $x_i \in \{0,1\}$ indicate that tile $i$ gets a high-resolution pass. Each high-resolution tile costs $c_i$ milliseconds on the target board. The gate solves

$$\max_{x} \sum_i x_i \, \hat{p}_i(s_t, d_{t-1}) \, w(\tau_i) \quad \text{s.t.} \quad \sum_i x_i c_i \le B - c_{\text{periph}},$$

The terms in this objective are defined as follows. $\hat{p}_i$ is a small learned predictor of the probability that a not-yet-resolved hazard lies in tile $i$, given telemetry $s_t$ (speed, yaw rate) and the previous frame's detections $d_{t-1}$. $w(\tau_i)$ weights tiles by how soon an object there would reach the vehicle at current speed, a short time-to-collision getting high weight. $B$ is the frame budget, for example 16.7 ms at 60 FPS. $c_{\text{periph}}$ is the cost of the peripheral pass. The objective is a cheap, closed-form stand-in for the value-of-observation idea in ActiveVOO. That work chooses what to sense by how much it advances a plan, and it reports success rising from 0.22 with exhaustive perception to 0.86 on visual ALFWorld. It computes that value with GPT-4o, however, which can never run on a Jetson or a microcontroller. We keep the principle and replace the model with a sub-millisecond predictor.

This differs from the two closest pieces of work. Attentional Bottleneck: Towards an Interpretable Deep Driving Network compresses features of pre-rendered bird's-eye rasters into a low-dimensional code to make driving decisions interpretable. It saves no compute, never sees raw pixels, and targets data-center hardware. We act on raw thermal frames and aim to cut wall-clock cost. Channel Gating Neural Networks skips computation per location based on a partial sum from the current input. Its 2.4x real speedup needed a custom ASIC. It is also purely bottom-up and has no way to take in vehicle state. Our gate is top-down, decided ahead of time, and coarse enough to speed up stock hardware.

Research Program

The build has three parts, all on a Jetson Nano and a Jetson Orin Nano with a FLIR ADK camera, the same hardware class as the reference system.

The first part is the detector pair. One detector runs on the whole downsampled frame and matches the reference pipeline: pruned YOLOv7-tiny, INT8. The second is a lightweight tile detector trained on native-resolution crops, and its detections are merged with the peripheral pass. Compression follows the standard recipe in Machine Learning for Microcontroller-Class Hardware: pruning plus integer quantization. Tile inference is batched so that the cost $c_i$ is measured on the board rather than estimated from floating-point operation counts.

The second part is the context gate. It is a small network, or a gradient-boosted model, mapping telemetry, the previous frame's detections, and a horizon estimate to per-tile hazard scores. The knapsack step above is solved greedily. Training labels come from the full-resolution detector run offline, and we mark which tiles contained objects that the peripheral pass missed.

The third part is the data. The public benchmarks carry no vehicle telemetry as far as the file reports, so we will collect our own logs. We will fit a FLIR ADK and a GPS/CAN logger to a retrofitted vehicle and drive fog, night, and clear routes at a range of speeds. Distant pedestrians, cars, and cyclists will be annotated at native resolution. For the public datasets we substitute a proxy context: ego-motion estimated from frame-to-frame optical flow plus the previous frame's detections. This proxy is itself an ablation, because it tests whether real telemetry adds anything beyond what the image stream already implies.

As a secondary study, we will feed the same gate scores to the in-cab display. Only hazards in high-weight tiles, meaning short time-to-collision in the current direction of travel, get visual emphasis. This responds directly to the reference system's field result: operators rated utility 3.7/5 but safety improvement only 2.8/5, citing screen distraction and clutter (Low-Cost Infrared Vision Systems). We treat this part as exploratory and measure it offline, as alerts per hour and the fraction of true short-time-to-collision hazards that get emphasized. A proper human study would come later.

Baselines

The naive reference point is the uniform full-frame pipeline of Low-Cost Infrared Vision Systems, a pruned and INT8-quantized YOLOv7-tiny run on every downsampled frame. It reports F1 0.8958 in foggy night and 90 FPS on the Jetson Orin Nano. We will retrain it on the same training split as our model, with the same augmentation and tuning budget, and run it on the same boards. Beating it means higher recall on objects under 16×16 pixels at native resolution, and earlier first detection of objects entering range, at equal or lower measured latency, with no loss of more than one point of F1 on large objects.

The second baseline is uniform high resolution. It runs the tile detector on every tile, which is the accuracy ceiling for our method, and we report its latency. It shows how much of the full-resolution gain the fovea recovers inside the budget.

The strongest existing competitor for the gating idea is bottom-up dynamic computation in the style of Channel Gating Neural Networks. We will implement it in two ways. One is the channel-gated detector itself, where we report floating-point savings and whatever wall-clock change the Jetson delivers. The other is a tile-level bottom-up variant that chooses high-resolution tiles by image saliency, meaning thermal contrast and temporal change, in the spirit of classic saliency models (Niebur et al. 1995). Both use the same per-frame budget $B$. The second variant is the fair head-to-head, because it changes only what drives the gate.

For accuracy reference on the public data, D3T reports 69.30 mAP on FLIR and 48.96 mAP on KAIST. That number comes from a domain-adaptation setting with a much heavier detector, so it bounds what large models reach rather than serving as an edge-budget competitor. For RGBT-Tiny, no reported scalar AP result was found on Lacuna. We will score it with the size-aware SAFit metric (C = 32) proposed there, because IoU collapses for tiny boxes: an 8×8 box offset by 2 pixels drops IoU from 1.0 to 0.39.

The key ablations isolate the top-down ingredient. The first is a tracking-only gate, where only $d_{t-1}$ is used. The second is a telemetry-only gate, where only $s_t$ is used. The third is the full gate with the $w(\tau_i)$ time-to-collision weighting removed.

Why not just…?

Why not just run the uniform detector, since it already exceeds the camera frame rate? On the Orin Nano, raw frame rate is indeed not the bottleneck; 90 FPS beats the camera's 60. The bottleneck is resolution. The reference network runs on a downsampled frame, and a distant hazard shrinks below detectability long before a vehicle at speed can stop. The uniform high-resolution baseline tells us how much latency full resolution would cost. If it fits the budget on the Orin Nano, the proposal's value there reduces to energy savings and the cheaper Jetson Nano, and we will say so.

Why not just track previous detections and crop around them? This is the real threat. Tracking covers objects already seen, and it may match our gate on persistent targets. It cannot pick up an object entering the scene at long range, which is exactly where speed-dependent context should help. The tracking-only ablation settles this, scored separately on newly appearing versus persistent objects. If telemetry adds nothing over tracking plus the peripheral pass, the top-down claim fails and the contribution becomes "foveation plus tracking."

Why not just use bottom-up saliency? In thermal fog imagery, warm clutter such as engines, road patches, and lights competes with pedestrians. A saliency-driven gate spends its budget on whatever is hottest, not on what lies in the vehicle's path. This echoes the finding that control networks carry almost no representation of task-irrelevant features (Dermody et al.). The matched-budget saliency baseline tests this directly.

Why not just filter alerts by persistence and ego-lane position? For the interface study this simple rule may be enough, and we include it as the display baseline. The gate earns its place there only if it emphasizes more true short-time-to-collision hazards at the same alert rate.

Proposed Experiments

The first experiment is decisive. On our own telemetry-logged fog, night, and clear drives, at a fixed measured budget of 16.7 ms per frame on the Jetson Nano, we compare the full gate against the uniform, saliency, and tracking-only baselines. We measure three things for objects entering the scene: recall on objects under 16×16 native pixels, distance at first confirmed detection, and time-to-collision at first detection. The hypothesis is supported if the full gate detects entering objects at greater distance than tracking-only and saliency, with non-overlapping bootstrap confidence intervals across drives. It fails if tracking-only matches it.

The second experiment tests transfer to public data. We run the gate with the proxy context on the FLIR dataset and on RGBT-Tiny, reporting SAFit-based AP and the per-object-size breakdown against uniform processing at matched latency. This checks whether the gain holds for tiny targets beyond our own vehicle.

The third experiment reports energy per frame and latency on both boards, separating time spent on the gate, the peripheral pass, and the tiles. Any saving must be real wall-clock time, not a floating-point count.

The final experiment is the offline alert-burden comparison against the persistence-and-lane rule. It is framed as groundwork for a later cab study rather than as a safety claim.