Pixel-Free Layout Priors for Tiny Thermal Object Detection

A thermal camera sees heat, not color. That makes it the right sensor for night, smoke and fog, but its data are scarce: few labeled thermal datasets exist, and they are hard to annotate. The problem is worst for objects seen from far away, such as a person or car from a drone, which cover fewer than 16×16 pixels. At that size a target is a faint warm blob with no internal detail. This proposal starts from a simple observation. Where objects sit relative to one another is the same in visible and thermal images, even though what they look like differs completely. People walk in loose groups. Cars line up at regular spacings. Cyclists appear near other road users at particular distances. We propose to learn this "scene layout" from abundant labeled visible-light images with a model that never sees a single pixel. We then use it to decide which faint thermal blobs are real objects and what they are.

Concretely, a small transformer reads only the set of candidate boxes a detector proposes: their positions, sizes, predicted classes and confidence scores. It is trained on visible-image annotations to fill in deliberately hidden objects and to reject deliberately inserted fake ones. Its inputs contain no image appearance, so it has no visible-to-thermal appearance gap to cross. The main claim is testable: with very few thermal labels, this appearance-free layout prior improves recall of tiny thermal objects more than either translating thermal images into pseudo-color images or enlarging the detector's view of surrounding pixels. We also expect the advantage to shrink as thermal labels become plentiful.

Where This Sits in Prior Work

Three lines of work attack thermal scarcity, and all three operate on appearance. Pixel translation turns thermal frames into pseudo-color images. Borrow From Anywhere runs a two-branch detector on the thermal frame plus a CycleGAN or UNIT-hallucinated color frame. PFGF fuses pseudo-visible generator features with thermal features and needs state-space filtering to suppress the generator's artifacts. Feature-level domain adaptation takes a different route. D3T trains separate visible and thermal teacher networks on an alternating schedule. SAGA desaturates object instances in visible images to remove color shortcuts. Both motivate our work, because they show the appearance gap between spectra is large and fragile to bridge. Both also constrain it: translation cannot invent texture that a 5-pixel blob does not contain. We transfer none of the appearance, only the arrangement.

Relational reasoning supplies the method. MBBR masks half the detected objects' visual features and reconstructs them from the other objects plus box geometry. This nearly doubled few-shot predicate recall on VRD (20.87% versus 10.41% Recall@20), which is evidence that masked reconstruction over object sets learns usable structure from little data. We reuse the masking idea but change what is reconstructed. MBBR predicts RGB visual features; we predict geometry and class, which carry over to thermal. RepGN links detection proposals in a graph whose edges come from box overlap (intersection-over-union, or IoU) and gains about one point of COCO mAP (36.4 to 37.5). Its edge definition cannot work for our targets. Tiny objects rarely overlap, and RGBT-Tiny shows that a 2-pixel shift on an 8×8 box drops IoU from 1.0 to 0.39. We therefore build attention between boxes from the Normalized Wasserstein Distance, a similarity that treats each box as a 2D Gaussian blob. RGBT-Tiny introduced this similarity for box regression, not for relations.

Finding Tiny Faces established that tiny targets must be recognized by looking around them rather than at them. Its context is pixel context from very large receptive fields, and it works because training and test share a modality. Our context is symbolic, made of other detected objects, and that is exactly what lets it be learned in one spectrum and used in another. COXNet is the closest system on tiny aerial visible-plus-thermal detection, but it needs paired, calibrated visible and thermal inputs at test time and models no object-to-object relations. We assume only a thermal camera at deployment.

Core Mechanism

A base thermal detector runs at a low confidence threshold and emits candidates $\{b_i, p_i, s_i\}$. Here $b_i$ is a box, $p_i$ a class-probability vector and $s_i$ a confidence score. Each box is encoded as a Gaussian with mean at its center and covariance set by its width and height. Attention between candidates $i$ and $j$ is biased by

$$ a_{ij} = \exp\!\left(-\frac{W_2(\mathcal{N}_i,\mathcal{N}_j)}{c}\right), $$

where $W_2$ is the 2-Wasserstein distance between the two box Gaussians and $c$ is a scale constant, following the normalized form in RGBT-Tiny. Unlike IoU, this similarity varies smoothly for disjoint boxes. The layout model $L$ outputs a revised objectness and class for every candidate.

Pretraining uses visible-image ground-truth boxes, with no images. Each scene is corrupted in two ways. We mask 50% of the objects, hiding their class and score. We also inject synthetic false boxes whose sizes and scores mimic a detector's clutter responses. $L$ is trained to recover the masked classes and to separate real boxes from injected ones:

$$ \mathcal{L} = \mathcal{L}_{\text{cls}}(\text{masked}) + \lambda\,\mathcal{L}_{\text{real/fake}}(\text{all}), $$

where $\lambda$ balances the two terms. At thermal test time, final scores combine the detector's score and $L$'s objectness through a scalar weight. That weight and $L$'s last layer are tuned on the small labeled thermal set. This small tuning step exists because the thermal detector's errors, such as warm rooftops and engine blocks, are distributed differently from the synthetic clutter.

Research Program

The first step builds the pretraining corpus from visible annotations that match the deployment viewpoint. These are the visible stream of RGBT-Tiny sequences held out from all thermal evaluation, the aerial views in IndraEye, and DOTA for vehicle layouts. Keeping the viewpoint matched matters, because layout statistics are viewpoint-specific. The second step trains the base thermal detector: ATSS with the SAFit loss, which RGBT-Tiny reports consistently improves AP over IoU and GIoU losses. We train it at thermal label budgets of 1%, 5%, 25% and 100% of RGBT-Tiny's thermal training sequences. The third step pretrains and calibrates $L$ as described above. The fourth step analyzes where the gains come from. We stratify recall by target size (under 8 px, 8–16 px, over 16 px), by the number of other objects in the frame, and by local thermal contrast. Contrast matters because low-contrast "thermal crossover" cases, where a target's temperature matches its background, are where appearance fails. Those are the cases where a layout prior should help most, as long as neighbors remain visible.

One honest risk shapes the design. Many RGBT-Tiny frames may contain a single object or a single class, which leaves little layout to exploit. Before any modeling, we will report the fraction of thermal test frames with at least three annotated objects. We will also report results separately for sparse and crowded frames rather than letting crowded scenes carry the average.

Baselines

The naive reference point is the SAFit-trained ATSS thermal detector at each label budget, with no visible data at all. Every other method is measured as a gain over it. The file lists no absolute RGBT-Tiny mAP for this configuration (no reported result found on Lacuna for ATSS+SAFit overall mAP on RGBT-Tiny), so we will run it ourselves.

The strongest existing competitor for using visible labels to help a thermal detector is D3T, whose dual teachers exploit labeled visible images and unlabeled thermal images. It reports 69.30 mAP on FLIR and 48.96 mAP on KAIST (D3T). No RGBT-Tiny number for D3T is available on Lacuna. We will run D3T on RGBT-Tiny's thermal stream with the same visible corpus, the same backbone, the same thermal label budget used as its thermal supervision, and the same number of GPU-hours for hyperparameter search. Because D3T and our method act on different parts of the pipeline, we will also report D3T plus our layout prior. We count as beating D3T either a higher AP on sub-16-pixel targets at the 1% and 5% budgets, or an additive gain when stacked on top of it.

The strongest pixel-translation competitor is MMTOD. On FLIR it reports 61.54 mAP against 53.97 for its thermal-only baseline, and it matches full-data performance with 25% of the thermal labels (Borrow From Anywhere). We will reproduce its 25%-data protocol on FLIR as a sanity check of our training setup, then run it on RGBT-Tiny at every budget. PFGF's numbers (47.1% mAP on FLIR, 67.3% on LLVIP) come from different protocols, as the gap to D3T's FLIR figure shows. They are therefore quoted, not compared directly.

Three ablations isolate the key ingredient. The first is a pixel-context head in the spirit of Finding Tiny Faces: a context crop about ten times the box size, fed through a small CNN and trained on the same thermal budget. It tests whether context helps because of symbolic transfer or merely because of context. The second is a count-based co-occurrence rescorer, which adjusts scores using class-pair distance histograms from the same visible corpus and involves no learning. It tests whether a transformer is needed at all. The third keeps our model but replaces the Wasserstein attention bias with RepGN-style IoU edges, which tests the edge definition. A fourth variant pretrains $L$ with no visible data, only on the thermal budget, to measure what the visible corpus contributes.

Why not just…?

Why not just translate thermal frames to pseudo-color and use a color-pretrained detector? This works for ground-level targets, as MMTOD's FLIR results show. However, PFGF found that generator artifacts must be filtered with extra machinery, and a sub-16-pixel blob has no texture for a translator to render faithfully. Our size-stratified comparison against MMTOD settles the question. We predict translation helps above 16 px and adds little below 8 px, where layout should help. If translation matches the layout prior on tiny targets, the core claim fails.

Why not just use domain adaptation such as D3T or SAGA? These align appearance features, and D3T's need for two separate teachers shows how costly that alignment is across spectra. They also cannot help when a target's appearance has vanished, as in thermal crossover. This alternative is a real threat at larger budgets and larger objects. The contrast-stratified results and the stacked D3T-plus-layout run will show whether the two are complementary or redundant.

Why not just give the detector a larger pixel context window? Finding Tiny Faces shows pixel context is powerful, raising WIDER FACE Hard AP to about 81–82% from 29–64% for prior methods. This is the most serious alternative. The difference is that pixel context in thermal must be learned from thermal labels, while symbolic layout can be learned from visible ones. Our prediction is therefore budget-dependent: the layout prior should win at 1–5% of thermal labels, and the pixel-context head should catch up or win at 100%. If the pixel-context head wins at low budgets too, the transfer argument is wrong and we would drop the method in favor of that head.

Why not just use simple co-occurrence statistics instead of a learned model? For sparse scenes a histogram may capture most of the signal. The count-based rescorer is included precisely to check this. The learned model justifies itself only if it beats the histogram on crowded frames, where relations involve more than pairs of objects.

Decisive Checks

The primary result is AP on sub-16-pixel targets on RGBT-Tiny's thermal test stream at 1% and 5% thermal labels. It is scored with both IoU-matched AP and SAFit-matched AP, since IoU matching is unstable at this size. The claim survives only if three things hold at those budgets: the layout prior beats the naive detector, the pixel-context head, and the co-occurrence rescorer; the gain concentrates in crowded and low-contrast frames; and it shrinks at 100% labels. A cross-viewpoint check tests whether the prior is really learning viewpoint-matched layout. We pretrain on ground-level visible scenes and evaluate on aerial thermal frames from IndraEye, and we expect a clear drop. FLIR at the 25% protocol serves as a ground-level control. There we expect small gains, because objects are larger and appearance already suffices.