Weakly supervised localization means learning where objects are, as boxes or masks, from labels that only say which objects appear in an image. It is the cheapest route to detectors for sensors nobody wants to annotate pixel by pixel, such as thermal cameras paired with ordinary color cameras on vehicles and robots. Two known problems meet in this setting. First, a network trained only to classify tends to highlight the single most telling part of an object, like a head or a warm torso, and misses the rest. Second, fusing two cameras can make results worse than using the better camera alone. A dark or glare-washed color image, or a thermal image full of hot pavement, pollutes the other stream.
The common fix for the second problem is a learned gate that decides how much to trust each camera. We argue that under image-level labels such a gate is trained by the wrong signal. Its only gradient comes from the classification loss, so it learns to route toward whatever makes the class easiest to recognize: the hottest pixel, the sharpest edge, or a background that happens to co-occur with the class. In other words, the gate reproduces and amplifies the part-domination problem. We propose instead to compute the per-location fusion weights from each camera's own calibrated evidence, with no gradient flowing into the gate. We then test whether separating the question "is this camera reliable here?" from "does using it here lower the loss?" closes more of the localization gap than any learned gate.
Core Claim
The claim has two halves, and both can be falsified. The diagnostic half says that on RGB-thermal data with image-level labels, an end-to-end learned spatial gate shrinks localization maps toward discriminative peaks and increases background bleed compared with the better single camera. We call this a weak-supervision-specific form of feature interference, and it is measurable as a fusion penalty on localization even when classification accuracy improves. The constructive half says that a parameter-free, stop-gradient gate removes this penalty and beats both single-camera and learned-gate fusion on localization. Such a gate is built from per-location evidential vacuity (how little evidence a camera's branch has at that spot) after scale-invariant standardization. The largest gains should appear at night, under spatially local corruption, and on frames containing no target.
Position Relative to Prior Work
Four groups of prior work bear on this claim.
The first group motivates the problem. Fusion penalties are well documented in fully supervised work. Residual Spatial Fusion Network shows that adding dark RGB features corrupts clean thermal features in segmentation. AFter loses tracking precision on RGBT234, falling from 90.1% to 86.0%, when its learned routers are replaced by fixed fusion. The fixes available there are either global scalar gates or routers trained with dense labels. Illumination-aware Faster R-CNN suppresses RGB to a 0.05 weight at night using one image-wide illumination value, and TNet scales thermal by a Retinex brightness score. Both require boxes or saliency masks, and both treat a camera as uniformly good or bad across the whole frame. Our setting removes the dense labels and asks for spatial rather than global gating. It is exactly the setting where their learned components lose their supervision.
The second group supplies the method we reuse. Beyond Magnitude and its companion SAEF show that fusion rules keyed to logit magnitude let a corrupted view with large logits hijack the decision. They fix this at the view level with instance-wise standardization and vacuity-weighted fusion, which needs no learned parameters. MIREL shows that instance-level uncertainty can be learned from bag-level labels alone, and that attention maps are poor stand-ins for uncertainty. We move SAEF's standardization from whole views to spatial locations. We use a MIREL-style argument to train per-location evidence from image labels. This turns out to be a combination neither paper attempts: SAEF is not spatial, and MIREL uses a single modality.
The third group supplies the localization backbone. SAOL replaces global average pooling with per-location class logits weighted by a spatial attention map. It raises CUB localization accuracy from 49.41% for standard class activation maps (CAM) to 52.39%. We give each camera its own SAOL-style head so that per-location evidence exists to be fused. Min-Entropy Latent Model and FI-WSOD are proposal-based detectors on single RGB images. We treat them as orthogonal refinements rather than competitors, because our question concerns what the fused map looks like before any proposal refinement.
The fourth group consists of the closest multimodal weakly supervised work, and it constrains our framing. MMCST fuses frozen CLIP text tokens with image patches and has no mechanism for an unreliable second modality. The histopathology segmentation work does filter irrelevant text with learned query tokens, but that filter is itself trained end to end on the segmentation objective, and its second modality is text rather than a spatially degraded sensor. SLAVC is the only multi-sensor weakly supervised localization study in the file. It relies on 0.9 visual dropout, reports 0.0% accuracy on objects smaller than 32² pixels, and exposes the "always-present" fallacy: models find a target even in frames that do not contain one. We adopt that paper's evaluation lesson by scoring on frames without targets.
Research Program
The model has two backbones, one for RGB and one for thermal. Each ends in a SAOL-style head that outputs per-location class evidence $e_m(p) \ge 0$ for modality $m$ at location $p$. Evidence here means non-negative support for each class, in the evidential deep learning sense. Each head is trained with its own image-level evidential loss, so every branch must stand on its own.
At each location, two steps produce the fused map. First, we standardize each branch's logits across locations within the image, giving $z_m(p)$. This removes the arbitrary scale differences that SAEF identifies as the cause of hijacking. Second, we compute the branch's vacuity $u_m(p) = K / S_m(p)$, where $K$ is the number of classes and $S_m(p)$ is the total Dirichlet evidence at that location. The fused per-location logit is
$$ Y(p) = \sum_m \mathrm{sg}\!\left[\frac{1-u_m(p)}{\sum_{m'} (1-u_{m'}(p))}\right] z_m(p), $$
where $\mathrm{sg}$ denotes stop-gradient. Because of the stop-gradient, the classification loss on the fused prediction can shape the branches but cannot push the fusion weights toward whatever lowers that loss.
Two design questions are tested explicitly rather than assumed. The first is whether cross-modal conflict should add to vacuity, since a location where standardized logits disagree sharply is a candidate for background bleed. The second is whether the weights should be computed on features or on logits. A fused-map vacuity also gives the model a way to abstain, which we use to score frames that contain no target.
The analysis program comes before the method. We will measure what a learned spatial gate actually learns under image-level labels, using three quantities: the entropy of the gate across locations, the ratio of predicted map area to ground-truth object area, and the fraction of gate mass that falls outside ground-truth regions. We measure these by day and by night, and before and after injected corruption. If learned gates turn out to be well behaved, the core claim is wrong and the proposal reduces to an efficiency argument. We state this in advance.
Baselines
The naive reference points are the thermal-only and RGB-only versions of the same SAOL-style localizer, together with naive feature concatenation before the head. The better single camera defines the fusion penalty. Any fused method that falls below it has been contaminated.
The strongest existing competitor is a learned per-location gate trained end to end on the image-level loss. It follows the router design of AFter, reduced to a single spatial-gating stage so that capacity is comparable to ours. This competitor matters most because it is what practitioners would build, and because the file's evidence of learned routing beating fixed fusion comes from fully supervised tracking. A second strong competitor is a global illumination gate in the style of Illumination-aware Faster R-CNN. We will train it both with the weak loss and with the day/night flags that KAIST provides, which gives it privileged information we do not use. A third is image-level SAEF fusion, which is scale-invariant but global. The fourth is late fusion of per-camera maps by averaging and by per-location maximum.
The ablation that isolates our key ingredient uses our exact gate formula but removes the stop-gradient, so that only the gradient path differs. A second ablation removes the standardization, which tests whether scale hijacking occurs spatially.
All methods share the backbones, input resolution, training schedule, and augmentation. Each gets the same hyperparameter search budget: our method searches only its vacuity and conflict settings, while the others search their gate learning rates and temperatures. Model selection uses a held-out split with image-level labels only. This avoids the ground-truth early stopping that SLAVC showed inflates prior results.
No reported result found on Lacuna for weakly supervised localization on KAIST or MFNet. The file's nearest numbers are fully supervised upper bounds: a 15.73% log-average miss rate on KAIST for Illumination-aware Faster R-CNN and 56.2% mIoU on MFNet for RSFNet. There is also the RGB-only weakly supervised gain of SAOL over CAM on CUB, 52.39% against 49.41%. Beating the learned gate means three things: higher localization quality at night on both datasets, no fusion penalty relative to the better single camera in any day/night split, and higher average precision on frames containing no target. These must hold across three seeds with non-overlapping intervals.
Why not just…?
Why not just use a global illumination gate? It is cheap, and on KAIST it may capture most of the day/night variation. It cannot, however, keep well-lit streetlight regions at night while discarding dark ones, and it cannot suppress hot pavement in otherwise clean thermal images. Those are precisely the local failures the file identifies as open. The illumination-gate baseline with privileged day/night flags, combined with our synthetic local-corruption test, will show whether local gating matters. If the global gate matches us on real data but not under local corruption, we will report that the benefit is limited to local degradations.
Why not just learn the spatial gate end to end, as AFter does? This is the real threat, and the proposal is built to settle it. AFter's gain from routing was measured with full box supervision. Under image labels, we predict the gate collapses toward discriminative peaks. The stop-gradient ablation isolates this directly. If the learned gate matches ours on map area and background-mass measures, our mechanism adds nothing.
Why not just take the per-location maximum of the two cameras' maps? A union fixes under-coverage for free, and a reviewer could reasonably expect it to win on recall. It also inherits every false positive from either camera, including thermal hits on warm backgrounds, which TNet documents. It also cannot abstain on empty frames. We expect the maximum to win on recall-heavy metrics and lose on precision and on frames without targets. The results will be reported separately so this trade-off is visible.
Why not just randomly drop whole modalities during training? Modality dropout makes each branch self-sufficient, but it does not tell the network which camera to trust at test time. SLAVC's heavy feature dropout comes at the cost of small objects. We include modality dropout as an extra baseline. We also apply it to our own branches to check whether it helps.
Proposed Experiments
The first dataset is KAIST, trained with image-level pedestrian-present labels. We evaluate CorLoc-style box localization from thresholded maps, pointing accuracy, and localization quality on small pedestrians, since SLAVC's collapse on small objects makes them a telling subgroup. Following SLAVC, we also report average precision over all frames including those without pedestrians, split by day and night.
The second dataset is MFNet, with PST900 as a transfer check. We derive image-level labels from the masks and evaluate the per-class mIoU of pseudo-masks thresholded from the maps, again split by day and night.
The third experiment is a controlled corruption suite applied at training and test time. It consists of local glare patches and occlusions in RGB and synthetic hot background blobs in thermal, at graded area fractions. We measure the fused-minus-best-single-camera gap as a function of corrupted area. This is where the prediction is sharpest: global gates and learned gates should cross below the single-camera line at small corrupted areas, while ours should not.
The first decisive check is the gate diagnostic. The end-to-end gate should show lower spatial entropy, smaller map-to-object area ratios, and more mass outside objects than our gate. The stop-gradient ablation alone should reproduce most of that difference.
The second decisive check is the empty-frame precision result.
The third decisive check is the scale stress test from SAEF, now applied per location. We multiply one camera's logits by factors from 0.01 to 100 and verify that our maps stay unchanged while the ablation without standardization degrades.
If the learned gate does not collapse in the first check, we will publish the diagnostic as a negative result and drop the method claim.
References
- Residual spatial fusion network for RGB-thermal semantic segmentation
- AFTER: Attention-Based Fusion Router for RGBT Tracking
- Illumination-aware faster R-CNN for robust multispectral pedestrian detection
- Does Thermal Really Always Matter for RGB-T Salient Object Detection?
- Beyond Magnitude: Scale-Invariant Evidential Fusion for Multi-View Classification
- Weakly-Supervised Residual Evidential Learning for Multi-Instance Uncertainty Estimation
- Spatially Attentive Output Layer for Image Classification
- Min-Entropy Latent Model for Weakly Supervised Object Detection
- FI-WSOD: Foreground Information Guided Weakly Supervised Object Detection
- Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization
- Multimodal Weakly Supervised Segmentation for Histopathology
- A Closer Look at Weakly-Supervised Audio-Visual Source Localization