When a camera goes dark at night, a multi-sensor perception system can do one of two things: ignore the camera and lean on what remains, or "hallucinate" a stand-in, for example by predicting camera-like features from a thermal sensor. Hallucination works on benchmarks. The best KAIST pedestrian detector in our file does exactly this. But it has two hidden requirements. It needs many perfectly aligned camera–thermal training pairs, and it treats every hallucinated pixel as equally trustworthy, even where thermal carries no information about what the camera would have seen. We propose to drop both requirements. The stand-in camera image comes from an unpaired image translator, a generator trained on separate piles of thermal and visible images with no alignment between them. That translator also outputs a per-pixel estimate of its own uncertainty. The fusion network then trusts each region of the synthetic camera image only as much as that uncertainty allows.
The claim is specific. A translator's own per-pixel uncertainty, split into an irreducible part (thermal cannot specify texture or color here) and a "model has not seen this" part, is a better reliability signal for weighting a synthesized modality than learned attention gates or feature-space distance scores. The second part of the uncertainty is also a good way to decide which few aligned pairs are worth collecting when pairing is expensive. The opportunity matters most in the regime that hallucination papers sidestep: plenty of unpaired single-sensor data, very little synchronized multi-sensor data.
What is known and what is missing
The motivating evidence comes from missing-modality work. Beyond Fusion trains a thermal-to-visible feature hallucination branch on paired daytime frames and fuses it through channel attention, reaching 5.01% log-average miss rate on KAIST (3.27% at night). Its hallucination is deterministic and paired. Probabilistic Conformal Distillation argues that deterministic reconstruction is fundamentally wrong when one modality cannot specify another ("information asymmetry"). It models the missing modality as a Gaussian in feature space, distilled from a teacher trained on complete paired data. This constrains our design: we must output a distribution rather than a point. It also leaves open our setting, because PCD needs a complete-modality teacher and never produces an image-space surrogate. Missing Modality Prediction for Unpaired Multimodal Learning handles unpaired image–text data, but its predicted features carry no uncertainty and it does not touch dense sensors.
The method comes from Robustness via Uncertainty-aware Cycle Consistency (UGAC). UGAC is an unpaired translator in the CycleGAN family, where two generators map back and forth and are trained so a round trip returns the input. It replaces the fixed L1 round-trip loss with a per-pixel generalized Gaussian likelihood whose scale and shape are predicted by the network. It adds Monte Carlo dropout (repeated stochastic forward passes) for model uncertainty. Its uncertainty maps correlate with actual translation error, and it is robust to input noise (area-under-SSIM 0.78 vs 0.60 for the best competitor on noisy Cityscapes). UGAC never asks whether those maps are useful to anything downstream. We test exactly that.
The consumer side comes from adaptive sensor weighting. Cocoon weights camera and LiDAR per object using conformal nonconformity scores, which measure how far a feature sits from class prototypes. It improves mAP by 15% over static 1:1 fusion under camera blackout. It also notes that global weighting ignores spatial heterogeneity, and its scoring adds roughly 0.5 s of latency. MoME routes queries among sensor-specific experts before decoding. It documents the "calibration trap": failing sensors produce confident wrong outputs, so trust must be judged from inputs or features, not from output confidence. SparseLIF learns per-modality uncertainty scores for fusion. All three assume real, paired sensor streams and only drop sensors at test time. None weights a synthesized stream by how well it could be synthesized.
Core mechanism
Let $t$ be a thermal image and $\hat v = G(t)$ the translated visible image. For each pixel $i$, the translator outputs scale $\alpha_i$ and shape $\beta_i$ of a generalized Gaussian. Its aleatoric (irreducible) variance has the closed form
$$ \sigma^2_{a,i} = \alpha_i^2 \,\frac{\Gamma(3/\beta_i)}{\Gamma(1/\beta_i)}, $$
where $\Gamma$ is the gamma function. Its epistemic (model) variance $\sigma^2_{e,i}$ is the variance of $\hat v_i$ across $K$ dropout passes.
A visible-domain backbone $f_V$ and a thermal backbone $f_T$ produce feature maps. The fused feature at location $j$ is
$$ F_j = f_T(t)_j + w_j \, f_V(\hat v)_j, \qquad w_j = \exp\!\big(-\lambda_a \bar\sigma^2_{a,j} - \lambda_e \bar\sigma^2_{e,j}\big), $$
where $\bar\sigma^2_{\cdot,j}$ is the pixel variance average-pooled to feature location $j$, and $\lambda_a, \lambda_e \ge 0$ are learned scalars. When the real visible image is present, a second weight from a lightweight quality estimator on the real image gates it in the same additive way. The design choice that matters is that $w_j$ is a fixed monotone function of input-side uncertainty with only two learned numbers. The detector therefore cannot learn to override it with confident output-side signals, which is the calibration trap MoME warns about.
The visible backbone $f_V$ can be pretrained on abundant unpaired visible data. This is the actual source of value. Translation lets knowledge learned from visible-only data reach a thermal-only input, which a thermal network trained on scarce pairs cannot obtain by itself.
Research Program
The first stage builds the translator. We train UGAC thermal↔visible on KAIST with pairing deliberately broken: daytime visible and thermal frames are shuffled across sequences. We then verify per-pixel calibration against the held-out true visible frames. The second stage trains the fused detector under simulated failure: visible blackout, random per-frame dropout, and additive noise. Following the Bernoulli failure model of the sensor-failure segmentation benchmark, we report expected miss rate across failure probabilities.
The third stage is the active learning question. Suppose a fixed budget of $B$ frames can be "paired", meaning the oracle reveals the synchronized visible frame and boxes. Which frames should those be? We compare random selection, maximum total uncertainty, maximum epistemic uncertainty, and maximum epistemic uncertainty restricted to the middle band of aleatoric uncertainty. The last is borrowed from the median-aleatoric client selection in ALFLTP, which found that extreme-uncertainty selection chases noise. We predict that aleatoric uncertainty marks regions no pair will fix, since thermal simply cannot specify them, while epistemic uncertainty marks regions a pair will fix. If that is right, ranking by total uncertainty should waste budget.
The fourth stage repeats the weighting experiment on DELIVER, where the RGB stream is synthesized from depth and event inputs. This tests whether the effect holds beyond one sensor pair.
Baselines
The naive reference points are two. The first is a thermal-only detector at test time. The second is the same two-stream detector trained with ModDrop, which randomly zeroes whole modalities during training. ModDrop is the cheapest way to make a network survive a missing sensor. On ChaLearn it held a Jaccard index of 0.680 under sensor drops versus 0.466 without it, against 0.881 with all channels. Both naive baselines see the same paired subset and the same unpaired visible pretraining for $f_V$, so any gain we show cannot come from extra data.
The strongest existing method is Beyond Fusion. Its reported figures are 5.01% overall and 3.27% night-time log-average miss rate on KAIST, and 16.62% on CVC-14, all trained on full paired data. We will reproduce it at full pairing to anchor our implementation. Our comparison is at reduced paired fractions of 1%, 5%, 20%, and 100%, where its hallucination branch gets only the pairs we get. For these low-pair regimes there is no reported result found on Lacuna for Beyond Fusion, so all such numbers are ours. Beating it means lower miss rate at 1–20% pairing under visible blackout, with no worse than its reproduced number at 100%.
For adaptive weighting, the strongest signal-level competitor is Cocoon-style conformal nonconformity weighting applied to the synthesized visible features. It uses the same translator and detector and replaces only $w_j$. We also include the adaptive-selection family represented by MAGIC++ in the failure benchmark for DELIVER. The file gives no numeric MAGIC++ score, only that it beats CMNeXt, whose mIoU falls from about 66% to about 50% without RGB. So no reported result was found on Lacuna for MAGIC++ mIoU, and we will rerun it.
The ablations that isolate our ingredient are five. The first uses the translator with $w_j \equiv 1$, meaning translation without trust. The second uses deterministic CycleGAN translation with a learned channel-attention gate. The third keeps aleatoric uncertainty only. The fourth keeps epistemic uncertainty only. The fifth is a free-form learned gate fed the uncertainty maps, which tests whether the monotone constraint matters. Every arm gets the same detector, schedule, and a fixed tuning budget of the same number of hyperparameter trials.
Why not just…?
Why not just fuse the real sensors adaptively and skip synthesis? This is the most serious threat. A synthesized visible image is a deterministic function of thermal, so in principle a thermal network could learn whatever the translator adds. Our argument is about data, not information. The translator carries a visible-domain prior learned from unpaired data that the thermal branch never sees. The paired-fraction sweep settles this. If ModDrop or MAGIC++-style selection matches us at every pairing level, translation adds nothing and we will report that. We expect any advantage to shrink as pairing approaches 100%, and the crossover point is itself a result.
Why not just use Beyond Fusion's deterministic hallucination with its learned attention gate? PCD's information-asymmetry argument predicts that deterministic hallucination invents structure where thermal is silent. MoME's calibration trap predicts that a learned gate will trust those inventions when inputs shift. Ablation two against our full model answers this directly. We will also test under input noise, where UGAC's robustness gap over vanilla cycle losses (0.78 vs 0.60 area-under-SSIM) should matter most.
Why not just model the missing modality as a distribution in feature space, as PCD does? PCD needs a complete-modality teacher trained on paired data, which is the resource we assume is scarce. Its Gaussian variance is not tied to identifiable image regions. We will run PCD at matched pairing fractions. If it wins at low pairing, the image-space route is unnecessary.
Why not just pick pairs at random or by maximum uncertainty? Random selection is surprisingly strong in many active learning studies, and ALFLTP shows maximum-uncertainty selection chases noise. We will report full miss-rate-versus-budget curves for all four selection rules. If random selection is within noise of our epistemic-band rule, we will drop the active-learning claim.
Decisive checks
The first check is whether translator uncertainty predicts downstream failure. Using pooled $\bar\sigma^2$ as a score, we measure AUROC for predicting whether a ground-truth pedestrian in that region is missed by the visible branch. UGAC established correlation only with pixel residuals. A low AUROC here kills the weighting idea before any detector comparison.
The second check is whether uncertainty weighting beats both "translate and trust everything" and "thermal only" under visible blackout at 5% pairing on KAIST. Gains must exceed seed variance over five seeds.
The third check is whether the aleatoric/epistemic split behaves as predicted. When pairs are added in high-aleatoric regions, those regions' uncertainty should stay roughly flat. When pairs are added in high-epistemic regions, uncertainty and miss rate should fall.
The fourth check is cost. We report inference latency for $K$ dropout passes against a single-pass, aleatoric-only variant, because Cocoon's 0.5 s overhead shows that reliability scoring can quietly make a system undeployable.
References
- Beyond Fusion: Modality Hallucination-based Multispectral Fusion for Pedestrian Detection
- Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness
- Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models
- Robustness via Uncertainty-aware Cycle Consistency
- Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion
- SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection
- Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness
- Privacy-Preserving and Uncertainty-Aware Federated Trajectory Prediction for Connected Autonomous Vehicles
- ModDrop: adaptive multi-modal gesture recognition