Video trackers that keep a memory of past frames, rather than one fixed snapshot of the target, handle rotation and occlusion well. They fail badly when the lighting changes. The cause is simple: each memory slot stores the target's shape and identity mixed together with the lighting it happened to be seen under. When a car drives from a lit street into darkness, the tracker compares a dark, noisy query frame against bright memories. The match collapses, and the corrupted frame is then written back into memory, which makes the next comparison worse. The scale of the problem is documented by NT-VOT211, which evaluates 43 trackers at night. Trackers that reach about 85% AUC (area under the success curve) on daytime TrackingNet fall to about 40% at night. The paper also reports that fine-tuning on night data yields only marginal gains.
We propose to split every memory entry into two parts. The first is a content part, holding geometry and identity, which is used for matching. The second is a compact style part, holding per-region lighting and color statistics, which is kept rather than discarded. At read time, the stored memories are re-rendered in the current frame's style before comparison. This amounts to a small unpaired image-to-image translation (mapping images between two visual domains without aligned before/after pairs) performed inside the tracker's feature space. The style part also tells the tracker whether a sudden change in the target is a lighting change, which is safe to absorb, or a content change such as an occlusion, which is unsafe to write into memory.
Core Claim
The obvious approach is to make features lighting-invariant: train with heavy photometric or translated-night augmentation so that style simply disappears. We argue this throws away information the tracker needs. Color and texture are often what separates the target from a look-alike distractor. Under drift, the right question is how the target looks now, not what it looks like with all appearance removed.
Our claim has two parts. The first is that a memory which stores content and style separately, and re-stylizes old entries to the query frame's conditions, beats both a conflated memory and an invariance-trained memory on night tracking. The second is that the gain should concentrate where invariance hurts, namely sequences with similar-looking distractors, while matching invariance elsewhere.
If an augmentation-only tracker matches us on distractor-heavy night sequences, the claim is false. The experiments are built to check exactly this.
Relation to Prior Work
Three memory trackers motivate the gap. MemTrack uses an LSTM-addressed memory and adds the retrieved memory as a gated residual to the first-frame template. STM stores past frames as key–value maps and reads them by dense pixel-to-pixel attention. STMTrack drops the fixed template entirely and keeps a six-frame space-time memory. All three store raw, entangled features and admit frames into memory without asking why the appearance changed. We keep STMTrack's memory structure and change what an entry contains and how entries are written.
The disentanglement literature supplies our methods. From ALADIN we take the observation that channel-wise mean and variance carry style but no spatial layout, so a style code built from them cannot smuggle geometry. From DSI2I we take the evidence that a single global style vector bleeds colors across regions, and its remedy of dense style with spatial shuffling. We compute style per coarse region, not globally, but we skip DSI2I's per-frame optimal transport because it is too slow for tracking. From Basak and Yin we take the paired contrastive heads: one forces the style code to identify the domain, and the other pulls the content of the same structure together across styles.
Harley et al. constrain the design. They show that jointly trained encoders "cheat" by hiding spatial information steganographically in the style stream. Freezing the backward network on generated inputs raises factor correlation from 0.695 to 0.998 on their synthetic benchmark. Our re-stylization loss is reconstructive, so it faces exactly this risk, and we adopt their uncooperative rule.
On the translation side, ToDayGAN shows that night-to-day translation of single images lifts localization on Oxford RobotCar from 13.9% to 52.9%. However, it runs per frame, as a separate model, with no temporal memory. CUT replaces the cycle-consistency requirement (translate forward and back to recover the input) with a patchwise contrastive loss, which avoids forcing a bijection when darkness destroys texture. We use CUT only offline, to generate night versions of labeled daytime training videos. It is never used at test time.
Research Program
Each memory entry for frame $i$ stores a content key $k_i$ and content value $v_i$, which are spatial feature maps, plus a style code $s_i$. The style code is the channel mean and standard deviation pooled over a coarse $4\times4$ grid of regions, shuffled across regions during training so that it cannot carry layout.
At read time, the query frame $q$ supplies its own style $s_q$. Each memory value is re-stylized by region-wise adaptive instance normalization (AdaIN, which replaces a feature map's per-channel mean and variance with target statistics) before the usual attention readout:
$$ \hat v_i = \sigma(s_q)\,\frac{v_i - \mu(v_i)}{\sigma(v_i)} + \mu(s_q), $$
where $\mu(\cdot)$ and $\sigma(\cdot)$ are the region-wise channel means and standard deviations, taken either from a feature map or read out of the style code. Keys are matched on content alone, so lighting does not change addressing. Values, which carry discriminative appearance, arrive in the current lighting.
The write gate uses both parts of the representation. It admits a new frame when content similarity to memory stays high, even if the style distance $\lVert s_t - s_{t-1}\rVert$ is large, because a large style distance alone means a lighting change. It rejects a frame when content similarity drops sharply, which signals an occlusion or a corrupted prediction.
Training uses only daytime labeled videos from GOT-10k and LaSOT. These are paired with night-styled copies produced by a CUT model trained on unpaired night imagery that is disjoint from NT-VOT211, such as the INIT night split. Because the boxes are known, the tracking loss applies to both versions. A content contrastive loss ties $k$ and $v$ across the day/night copies. A style contrastive loss forces $s$ to separate the two conditions. A re-stylization loss asks $\hat v$ computed from a day memory with night style to match the feature map of the night frame.
Following Harley et al., the re-stylization loss updates only the encoder on real inputs and freezes it on translated ones, so the content path cannot learn the translator's hidden codes. Per-video, fixed-seed translation avoids frame-to-frame flicker in the synthetic sequences.
Baselines
The naive reference point is STMTrack exactly as released, trained on daytime data. It reports 0.642 AO (average overlap) on GOT-10k and 80.3% AUC on TrackingNet (STMTrack). We found no reported result on Lacuna for STMTrack on NT-VOT211, so we will run it ourselves.
The strongest existing competitor on night tracking is ProContEXT, the top tracker in NT-VOT211's study at 40.08% AUC and 54.54% precision (NT-VOT211). We will report that published number and also rerun it under our protocol. Beating it means exceeding 40.08% AUC on NT-VOT211 without NT-VOT211 training data.
The most dangerous baseline is invariance training. This is STMTrack trained on the same day videos plus the same CUT-generated night copies, together with Fourier low-frequency style swapping and color jitter. It uses identical architecture size, data, epochs and hyperparameter search budget, differing from ours only in the absence of a style stream. The test that matters is whether we beat it on NT-VOT211 overall and, specifically, on its similar-distractor attribute subset. Matching it overall but losing on distractors would falsify our reasoning.
A pipeline baseline places ToDayGAN or CUT as a per-frame night-to-day preprocessor in front of unmodified STMTrack. We report both accuracy and frames per second for it.
Ablations isolate each ingredient. One removes re-stylization while keeping the split memory. One replaces the write gate with STMTrack's default schedule. One drops uncooperative training, where we expect style leakage measurable by decoding box coordinates from $s$. One swaps the regional style code for a single global vector, to test style bleeding.
Why not just…?
Why not just fine-tune an existing tracker on night videos? NT-VOT211 reports that this yields only marginal gains. Our invariance baseline is a stronger version of this idea, because it supplies far more night data through translation. If it closes the gap, the cheaper answer wins, and we will report that.
Why not just translate each frame to daytime first? ToDayGAN's gains come from single-image retrieval, where flicker is irrelevant. In tracking, per-frame translation adds a full generator to each frame and introduces boundary jitter. The CUT paper shows CycleGAN-style translators distorting structure, with an FID (Fréchet Inception Distance, lower is better) of 77.2 against CUT's 45.5 on horse→zebra. We measure the preprocessor directly on NT-VOT211 accuracy and speed rather than assuming it fails.
Why not just make features lighting-invariant with augmentation? This is the real threat. It may capture most of the benefit. Our answer is an empirical bet: invariance removes the color and texture cues that separate targets from distractors, while re-stylization keeps them. The distractor-subset comparison settles it either way.
Why not just use a global AdaIN style vector? DSI2I documents style bleeding from global vectors, and Harley et al. show low-dimensional style codes cannot capture complex illumination. Headlight glare is local, so a single code would re-light the whole memory with one lamp's color. The global-vector ablation tests this.
Decisive Checks
The primary result is NT-VOT211 AUC and precision, alongside GOT-10k AO and TrackingNet AUC to confirm that daytime performance is not sacrificed. Two diagnostic checks back it up.
The first measures memory corruption. We count the fraction of memory slots whose stored box has IoU (intersection over union with the ground-truth box) below 0.5, plotted against time-since-illumination-change. We expect our gate to keep this fraction flat where STMTrack's rises.
The second is a leakage probe. A linear decoder tries to recover target position from the style code, with and without uncooperative training. Near-chance decoding is required before any accuracy gain is credited to disentanglement rather than to a hidden second content path.
References
- NT-VOT211: A Large-Scale Benchmark for Night-Time Visual Object Tracking
- Learning Dynamic Memory Networks for Object Tracking
- Video Object Segmentation Using Space-Time Memory Networks
- STMTrack: Template-Free Visual Tracking With Space-Time Memory Networks
- ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similarity
- DSI2I: Dense Style for Unpaired Exemplar-based Image-to- Image Translation
- Semi-supervised Domain Adaptive Medical Image Segmentation Through Consistency Regularized Disentangled Contrastive Learning
- Image Disentanglement and Uncooperative Re-Entanglement for High-Fidelity Image-to-Image Translation
- Night-to-Day Image Translation for Retrieval-based Localization
- Contrastive Learning for Unpaired Image-to-Image Translation