When a detector that fuses a visible (RGB) camera with a thermal camera is deployed on a new target class or under a new operating condition, the main thing that has to change is often not what the object looks like. It is which sensor to trust, and at which image scale. A tiny vehicle at night may survive only in the thermal channel's fine-resolution features. The same vehicle at noon during thermal crossover, when object and background reach the same temperature, may survive only in RGB. A camera rig whose two sensors are a few pixels out of alignment makes thermal features unreliable at fine scales but harmless at coarse ones.
We propose to treat this trust pattern as the thing to adapt from a handful of labeled image pairs. Each level of the detector's feature pyramid gets a small learnable modality gate. Gradient-based meta-learning, which trains a starting point such that a few gradient steps on a small labeled support set give good task-specific parameters, learns the gates' initialization and step sizes across many simulated adaptation episodes. At test time, only the gates move.
The bet is that adapting a few hundred gate parameters per pyramid level is the right amount of freedom. It is enough to re-route evidence toward the modality and scale where a small target is still visible. It is too little to overfit five examples or to erase what the fusion network already knows. It is also cheap enough that backpropagating through the inner adaptation loop stays tractable on a dense multi-scale detector.
Core Claim
In few-shot adaptation of an RGB-thermal detector, a meta-learned initialization of per-scale cross-modal gates should beat three alternatives at 1 to 5 shots: fine-tuning the same gates from a conventionally trained initialization, adapting the whole fusion network, and predicting the gates directly from the input image without labels. The gain should concentrate on small objects and on conditions where the dominant modality fails (night for RGB, crossover for thermal, misalignment at fine scales).
Research Program
The base detector follows the multispectral detectors in the file. Two backbones produce RGB and thermal feature pyramids. At each pyramid level $l$, an RGB-queries-thermal cross-attention block produces a thermal context map $\tilde T_l$, and the fused feature is
$$F_l = \sigma(\gamma_l) \odot \tilde T_l + \big(1-\sigma(\gamma_l)\big) \odot R_l,$$
where $R_l$ is the RGB feature at level $l$, $\gamma_l$ is a per-channel gate-logit vector, $\sigma$ is the logistic function, and $\odot$ is channel-wise multiplication. Fused levels then pass through a top-down pyramid into an oriented-box head.
The gating form is borrowed from gated cross-attention in An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention, which uses a learnable gate so that a dominant stream cannot wash out a secondary one. Our change is to make the gate scale-specific and to make it the only quantity adapted at test time.
Training is bi-level. An episode samples a task: a held-out class or condition slice, with $K$ labeled support pairs and a query set. The inner loop takes a few steps on the gates only, $\gamma' = \gamma - \alpha_l \odot \nabla_\gamma \mathcal{L}_{\mathcal S}(\theta,\gamma)$, where $\theta$ is all other parameters and $\alpha_l$ is a learned per-level step size. The outer loop updates $\theta$, the gate initialization $\gamma$, and $\alpha$ using the query loss after adaptation. This is the bi-level structure described in Meta-Learning: A Survey. Restricting the inner loop to $\gamma$ is how we address two problems in the file. The first is the inner-loop forgetting that Multi-Modal Fusion by Meta-Initialization reports, where support-set gradient steps wash out cross-modal guidance. The second is the memory cost of second-order gradients through dense pyramids. The second-order term only couples $\theta$ to a low-dimensional $\gamma$, and we will report peak memory to check that this actually helps.
Episodes are built from three sources. The first is class hold-out on DroneVehicle (Drone Based RGBT Vehicle Detection and Counting: A Challenge): five vehicle classes, of which one or two are held out as novel. The second is condition slices: day and night from the illumination annotations, plus a crossover-heavy subset. The third is synthetic sensor offsets, whose evaluation counterpart is the misaligned DVMA benchmark from Bridging Severe Cross-Modal Misalignment. That paper's "downsampling trap" observation, that pooling hides small misalignments at coarse levels, predicts a specific pattern: under offset, adapted gates should drop thermal at fine levels and keep it at coarse ones. We will check this directly by plotting adapted $\sigma(\gamma_l)$ against level and offset magnitude. Cross-dataset transfer uses M3FD and FLIR from TFDet: meta-train on DroneVehicle and adapt to their classes.
We position the work against the vision-language meta-learners. Multimodal Few-Shot Object Detection with Meta-Learning Based Cross-Modal Prompting meta-learns language prompts from support images for RGB detection, and it explicitly reports failures on very small objects. Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection adds visual prototypes to GroundingDINO because text cannot describe domain appearance. Adaptive Cross-Modal Few-shot Learning mixes visual and word-embedding prototypes with a class-wise coefficient. All three use language as the auxiliary modality and adapt what the class looks like. We use a physical second sensor and adapt how the two sensors are weighted at each scale. AM3's finding that its mixing coefficient shifts toward visual evidence as shots increase motivates letting the trust weight depend on the task. We push that idea down to individual pyramid levels and learn it by gradient.
Baselines
The naive reference is the base-trained fused detector applied to novel tasks with no adaptation. When classes are novel, its head gets a minimal update: only the novel-class rows of the classifier are trained, so that it can output the new class at all. Any method that fails to beat this has learned nothing from the support set.
The strongest practical competitor is plain fine-tuning of a conventionally trained detector. We take the same architecture trained with standard supervised learning on base tasks, then fine-tune on the $K$-shot support set, sweeping three parameter scopes: head only, gates plus head, and everything. Few-shot detection has repeatedly found that simple fine-tuning is hard to beat, and Multimodal Few-Shot Object Detection with Meta-Learning Based Cross-Modal Prompting itself gains from adding gradient fine-tuning (MSCOCO 1-shot AP rising from 5.6 to 6.5 with DeFRCN). The "gates plus head from a non-meta initialization" variant is the key ablation. It uses the same adapted parameters and the same inner-loop budget, and differs from our method only in whether the starting point was meta-learned. All variants get the same base data, the same number of adaptation steps, and an equal hyperparameter search budget (learning rate and steps) tuned on a held-out validation set of tasks, never on test tasks.
The strongest label-free competitor is input-conditioned gating. A small network predicts $\gamma_l$ from global image statistics and RGB illumination, in the spirit of UA-CMDet's uncertainty-aware weighting and JFRDet's illumination-guided fusion, and is trained on all base data. It needs no support labels, so it is the right test of whether labels add anything.
The metric-based competitor is AM3-style prototype matching (Adaptive Cross-Modal Few-shot Learning) on region features from the RGB and thermal pyramids. A learned mixing coefficient between the two sensors' prototypes is placed on the same base detector.
Full MAML over all fusion parameters tests whether restricting the inner loop matters. A single global gate, meta-learned the same way, isolates the contribution of per-scale gating.
For reference ceilings, only full-data numbers exist. UA-CMDet reports 64.01% mAP on DroneVehicle against 47.91% RGB-only and 59.15% IR-only (Drone Based RGBT Vehicle Detection and Counting: A Challenge), and JFRDet reports 69.7% mAP50 on DVMA (Bridging Severe Cross-Modal Misalignment). No reported result was found on Lacuna for few-shot novel-class detection on DroneVehicle, DVMA, M3FD or FLIR, so we will establish the few-shot baselines ourselves. Beating the field means outperforming the best fine-tuning scope and the input-conditioned gate by a margin larger than the standard deviation across at least five random support-set draws. The margin must appear on novel-class mAP50 for small objects at 1 and 5 shots, without lowering base-class mAP by more than one point.
Why not just…?
Why not just fine-tune a pretrained RGB-T detector on the few shots? This is the main threat, and the proposal is built so that it is tested directly rather than argued away. The "gates plus head from standard initialization" baseline adapts the same parameters with the same budget. If it matches our method, then meta-learning adds nothing and the useful contribution reduces to "adapt only per-scale gates," which is still worth reporting. We expect meta-learning to help mainly at 1 to 2 shots, where gradient steps from an unprepared starting point are noisy. The episodic fine-tuning study (Episodic fine-tuning prototypical networks for optimization-based few-shot learning) shows naive test-time fine-tuning dropping ESC-50 accuracy from 83.95% to 66.50%, while meta-structured adaptation reached 88.36%. That is audio classification, so it motivates the hypothesis rather than settling it.
Why not just use metric prototypes with an adaptive modality mixture? It is cheaper and has a strong record. On iNat-Anim, the AM3 baseline (80.8% 1-shot) actually beat the gradient-based FuMI (78.9%) (Multi-Modal Fusion by Meta-Initialization). That result constrains us: gradient-based adaptation of a classifier head is not automatically better. Our argument is narrower. Prototype matching operates on pooled region features, after downsampling has already erased small thermal or RGB signatures, whereas per-scale gates change which evidence survives into the pyramid. The decisive comparison is therefore on objects below roughly 16 pixels. If the AM3-style baseline ties there, we will concede the point.
Why not just predict the gates from the image, with no labels? Illumination is observable from the input, so for day/night shifts a feedforward gate may suffice, and we expect it to be competitive on those slices. Labels should matter when the right trust pattern depends on the target class or on unobservable rig properties. Examples are which sensor shows a novel vehicle type at small scale, or how a particular rig's misalignment affects fine levels. We report results separately on illumination-only, crossover, misalignment and novel-class tasks. If the input-conditioned gate matches us everywhere, the claim that support labels carry this information is false.
Why not just upsample or tile the input? Generalized-Scale Object Counting with Gradual Query Aggregation reports 25.33 GB versus 4.98 GB of memory for brute-force upsampling against gradual aggregation, and tiling splits objects at tile edges. Inside a bi-level loop that cost is multiplied by the number of inner steps. We include a 2x-upsampled fine-tuning baseline at matched memory to quantify this.
Proposed Experiments
The primary experiment runs 1-, 2-, 3- and 5-shot novel-class episodes on DroneVehicle, with three class hold-out splits and five support draws each. We report mAP50 overall and by object size, plus base-class mAP after adaptation to measure forgetting. The second experiment uses condition-shift episodes: night, crossover, and DVMA misalignment, adapted from $K$ labeled pairs of the new condition. The third is cross-dataset adaptation to M3FD and FLIR classes.
There are three decisive checks. First, meta-initialized gates must beat identically adapted gates from a standard initialization, with the gap shrinking as shots increase; a gap that does not shrink would suggest the improvement comes from the initialization being better trained rather than better placed for adaptation. Second, per-scale gates must beat a single global gate, with the difference concentrated on small objects. Third, under DVMA offsets, the adapted fine-level gates must move away from thermal while coarse-level gates do not. This is the signature predicted by the downsampling-trap analysis, and its absence would mean the gates are not doing what we claim even if mAP improves.
We will also report peak GPU memory and adaptation time against full MAML. Restricting the inner loop is only worth its added design complexity if full MAML is substantially more expensive at no better accuracy.
References
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Meta-Learning
- Multi-Modal Fusion by Meta-Initialization
- Drone Based RGBT Vehicle Detection and Counting: A Challenge
- Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration
- TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection
- Multimodal Few-Shot Object Detection with Meta-Learning Based Cross-Modal Prompting
- Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection
- Adaptive Cross-Modal Few-shot Learning
- Episodic Fine-Tuning Prototypical Networks for Optimization-Based Few-Shot Learning: Application to Audio Classification
- Generalized-Scale Object Counting with Gradual Query Aggregation