Unpaired image translation turns images from one camera type into another, for example thermal into visible, using two unrelated image collections and no aligned pairs. It would let trackers trained on ordinary visible video run on thermal footage without retraining. It breaks down on video for two reasons. The generators that preserve fine detail do so through skip connections, which copy raw early feature maps from the input side of the network straight to the output side. Those maps carry source-camera artifacts such as thermal halos and sensor noise into the output. And every frame is translated independently, so small motions and passing occluders make object boundaries flicker, bleed, and fragment. We propose one spatial bypass that addresses both problems. Instead of copying feature values, the generator passes only pooling switches: the positions, inside each small pooling window, that the encoder selected when it downsampled. The decoder uses these positions to place activations back where they came from. Across frames, the switches are treated as a hidden state. A small motion model predicts where each window's selection should move, and that prediction is fused with the current frame's evidence, much as a tracker predicts an object's position through an occlusion.
Stated in two sentences: pooling switches carry "where" without carrying "what", which makes them a geometry channel that should leak less source-domain texture than skip features. Because raw switches jump discontinuously under tiny motions and at occluder edges, we filter them over time with a predictive state model, and that filtering should remove translation flicker and boundary bleed without any optical flow at test time.
Core Claim
The claim has two testable parts. The first is that an unpaired generator whose only spatial bypass is index-guided unpooling can keep thin thermal-to-visible boundaries about as well as a U-Net skip generator while transferring measurably less thermal texture. The second is that per-frame max-pool switches are a major, previously unnamed source of temporal flicker in translated video, and that propagating them with a predictive state model reduces flicker and occlusion-boundary errors more than either shift-equivariant sampling or flow warping.
Relation to Prior Work
Three lines of work supply the index mechanism, and all of them are single-frame. Learning Deconvolution Network for Semantic Segmentation routes decoder activations through recorded max-pool switches to recover thin structures such as chair legs, reaching 72.5% mean intersection-over-union (mIoU) on PASCAL VOC 2012 in ensemble. Learnable Polyphase Sampling learns which subgrid to keep and inverts that choice when upsampling, which gives exact equivariance to circular shifts. MergeVQ stores a token-merge assignment that lets the decoder unmerge tokens back to pixel positions, reaching rFID 0.54. We reuse the switch-routing idea from these papers. Our contribution is to make the switches a state that evolves over time.
Learnable polyphase sampling also constrains the design. It shows that fixed switches are sensitive to shifts. It fixes this only for global translations, though, and not for independent object motion or occlusion. That limit is exactly the regime we target, so the method is a baseline and not a solution.
Two tracking papers motivate the predictive component. InstMove predicts instance masks with a recurrent model driven by appearance-free mask motion and beats RAFT optical flow by a wide margin under occlusion (67.8 vs 46.3 mAP on OVIS-Sub). That result is our main evidence that a learned motion prior, not flow, is the right way to carry geometry through occluders. InstMove still sharpens boundaries with ordinary skip features, and it never translates between camera types. MambaTrack shows that learned state-space predictors fed with their own predictions during occlusion outperform Kalman filters. It predicts only box coordinates and has no dense decoder.
On the translation side, CUT keeps content by contrasting corresponding patches, and Patch Your Matcher trains a U-Net translator whose discriminator checks the geometry of correspondences found by a frozen matcher. Both are static-image methods. Both also rely on value-carrying features, patches in CUT and skips in the U-Net, which is where we believe thermal texture leaks through.
Research Program
The generator is a CUT-style encoder-decoder. Each downsampling stage uses max-pooling with recorded switches, and the matching decoder stage uses unpooling followed by learned convolutions to fill in the sparse map. No feature values cross from encoder to decoder except through the bottleneck. We train it on VTUAV thermal and visible frames drawn from different sequences, so the training data is genuinely unpaired. The losses are CUT's patchwise contrastive loss plus an adversarial loss.
For video, each pooling window at location $w$ and time $t$ holds a distribution $p_t(w)$ over its $k$ candidate positions ($k=4$ for 2×2 pooling), instead of a hard argmax. The update is
$$p_t(w) \propto \hat p_{t|t-1}(w) \odot q_t(w),$$
where $q_t(w)$ is a temperature softmax of the current encoder activations inside the window. The prior $\hat p_{t|t-1}(w)$ is $p_{t-1}$ moved by a coarse motion field. That field comes from an InstMove-style recurrent predictor that runs on instance masks, which we obtain by running a frozen segmenter on the translated output. The decoder unpools with the posterior $p_t$. Near-hard posteriors reproduce ordinary unpooling. Windows where the current evidence is ambiguous, such as straddled occluder edges or motion-blurred regions, lean on the prediction. The predictor is trained with a self-supervised loss: it must forecast the next frame's switch distribution from past frames. That loss uses only thermal video, so the whole method stays unpaired.
We then analyze two things. We measure how often raw switches flip between consecutive frames, and whether those flips line up with frames that show flicker. We also measure how much thermal-specific signal each bypass type transfers.
Baselines
The naive reference is per-frame CUT with its standard generator, applied independently to each thermal frame. It is the method a practitioner would reach for first, and it isolates the cost of doing nothing temporal.
The strongest translation competitor is a U-Net skip generator trained with the same losses, in the style of Patch Your Matcher's generator. Patch Your Matcher's correspondence-aware discriminator is added where feasible. This generator is also given a flow-warped temporal consistency loss during training, because that is the obvious way to make a skip generator temporally stable. We found no reported result on Lacuna for unpaired thermal-to-visible video translation on VTUAV, and Patch Your Matcher's numbers are not in the file. We will therefore train every translator ourselves on identical unpaired splits, at identical resolution, with the same GPU-hour budget and the same hyperparameter-search budget. Beating this baseline means lower temporal warping error and lower thermal leakage at equal or better boundary F-score.
The strongest downstream reference is HMFT from VTUAV, which fuses real registered visible and thermal video and reports a maximum success rate (MSR) of 62.7% and a maximum precision rate (MPR) of 75.8%. We run HMFT with translated thermal substituted for its visible input, and separately with thermal duplicated into both inputs. The duplicated-thermal run is the lower bound. The reported 62.7% MSR with real visible input is the ceiling. A result counts as success if it closes a substantial fraction of that gap and beats the U-Net baseline's substituted-input score.
Three ablations isolate the key ingredient. The first uses hard per-frame switches with no temporal filter. The second replaces max-pool switches with learnable polyphase sampling. The third replaces the learned predictor with RAFT-warped switch distributions.
Why not just…?
Why not just use U-Net skip connections with a flow-based temporal loss? This is the real threat. It may match us on flicker, because the temporal loss targets flicker directly. Our bet rests on the leakage question instead. Skip features carry source values, while switches carry only positions. The noise-hiding behavior that CUT describes for cycle-based models suggests that value channels get exploited. If the U-Net baseline ties us on both leakage and flicker, the index bypass has no justification, and we will report that.
Why not just use learnable polyphase sampling, which is already shift-equivariant? Learnable Polyphase Sampling guarantees consistency under global circular shifts. A drone video contains camera motion, which is close to a global shift, and object motion and occlusion, which are not. We split the evaluation into frames dominated by camera motion and frames dominated by object motion or occlusion. We expect polyphase sampling to match us on the first split and lose on the second. If it wins on both, the temporal filter adds nothing.
Why not just warp the switches with optical flow? InstMove's 67.8 vs 46.3 mAP gap on OVIS-Sub shows that flow collapses under occlusion. Flow also has to be computed on the thermal input at test time, and thermal is exactly where flow is weakest. The RAFT-warping ablation tests this directly on our data.
Why translate at all instead of training a thermal tracker? When labeled thermal video exists, native training may simply win. Translation is worth doing in the regime where a strong visible-light tracker exists and target-modality labels do not. The duplicated-thermal HMFT run shows how much the translated stream adds over thermal alone.
Proposed Experiments
All experiments use VTUAV. Its 24.4k instance masks provide ground-truth visible-light geometry for measuring boundary F-score. We compute boundary F-score by running a frozen segmenter on the translated frames and scoring the result against these masks. Flicker is measured as the warping error between consecutive translated frames, with flow computed on the real registered visible frames, which are held out and used only for evaluation. Leakage is measured by the accuracy of a thermal-versus-visible classifier on translated patches within a few pixels of object boundaries. Distribution quality is measured with FID. Downstream tracking is scored as MSR and MPR with HMFT's input substituted as described.
A controlled stress test pastes moving thermal crops of other objects across tracked targets. Because the occluder is synthetic, its exact geometry is known, so we can measure boundary bleed at the occluder edge as a function of how long the occlusion lasts.
Two checks decide the outcome. The first is whether switch-flip rate correlates with per-frame warping error in the unfiltered model; if it does not, the flicker hypothesis is wrong. The second is whether the filtered model beats both the flow-regularized U-Net and polyphase sampling on the object-motion and occlusion split while staying within noise of them on boundary F-score. A clean negative on either check is informative in its own right. It would show that switch-level geometry is not the bottleneck in video translation.
References
- Learning Deconvolution Network for Semantic Segmentation
- Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks
- MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization
- InstMove: Instance Motion for Object-centric Video Segmentation
- MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Model
- Contrastive Learning for Unpaired Image-to-Image Translation
- Patch Your Matcher: Correspondence-Aware Image-to-Image Translation Unlocks Cross-Modal Matching via Single-Modality Priors
- Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline