Telling Shadow from Surface: Non-Local Illumination Fields for Color-Histogram Tracking

Color histograms count how often each color appears in a region. They are among the cheapest and most rotation-tolerant appearance models in visual tracking, and they fail in a predictable way. When a target walks from sunlight into shade, or past a window into tungsten light, its colors shift, and the tracker can no longer tell it from the background. The standard fixes either assume one light for the whole image (global white balance) or correct each pixel using only its immediate neighborhood. We argue that the second fix runs into a basic ambiguity. Within a small window, a reddish patch could be a red shirt under white light or a gray shirt under red light, and nothing local separates the two. What does separate them is evidence from far away: surfaces of different color lit by the same source shift in the same direction. A model that sees only a local window cannot gather that evidence.

The proposal is to estimate a smooth, spatially varying illumination map over the tracker's search region with a lightweight long-range attention module, and to compute the tracker's color likelihood on illumination-corrected colors. The illumination estimate pools evidence along entire rows and columns of the region rather than within a fixed kernel. The tracker should then separate light from surface under shadows and mixed lighting while keeping real-time speed. The question we want answered is narrow and testable: how large a receptive field does illumination correction need before a color-histogram tracker stops confusing light with surface, and does a long-range module beat a local one with matched capacity?

Core Claim

Our claim is that the failure of color likelihood maps under non-uniform light comes from the receptive field, not from the histogram itself. Local normalization, such as dividing each pixel by a blurred local average, removes the target's own color whenever the target fills the window. Global correction cannot follow a shadow edge. An illumination field estimated with long-range spatial pooling, in a color space where lighting changes become additive shifts, should avoid both failures. Plugged into an existing target-aware correlation filter, it should improve robustness on illumination-variation sequences without losing ground on background-clutter sequences.

The work builds on two pieces of prior work. Convolutional Color Constancy shows that in log-chrominance coordinates, $u=\log(G/R)$ and $v=\log(G/B)$, a change of illuminant color becomes a 2D translation. We reuse this coordinate system, but we let the translation vary over space, whereas CCC applies one translation to the whole image. CCNet introduces criss-cross attention, in which each pixel attends only to its own row and column; applying it twice gives every pixel a full-image receptive field. CCNet reports 11x less GPU memory and 85% fewer FLOPs than a full non-local block. We reuse this module as the estimator.

Research Program

The tracker host is the target-aware correlation filter of Learning target-aware correlation filters for visual tracking, called TACF. A correlation filter tracker learns a template that responds strongly at the target's location in each new frame. TACF weights its filter with a per-pixel target-likelihood map, the product of a box prior and a Bayes color-histogram score. Its authors state that this map fails under similar-colored backgrounds and sudden illumination change, which is exactly the failure we target. We change only how the color score is computed, so any gain can be attributed to that change.

For each frame, the search region is downsampled to a coarse grid, roughly 32×32. A small network then predicts an illumination offset $\ell(x)\in\mathbb{R}^2$ at each grid cell $x$, expressed in log-chrominance and upsampled smoothly. The network has a few convolution layers followed by two shared-weight criss-cross attention passes. The tracker's color histogram is then scored on corrected colors:

$$\tilde{c}(x) = c(x) - \ell(x),$$

where $c(x)=(u,v)$ is the observed log-chrominance at pixel $x$. Under a multiplicative light model, $\tilde{c}$ depends only on surface color.

One online signal regularizes the estimator. Background pixels that were tracked confidently in the previous frame have a known corrected color. Their frame-to-frame chromaticity change is therefore a direct, if noisy, reading of the illumination change at those locations. These readings enter the attention layers as extra input channels. This anchoring is what lets distant background surfaces inform the correction applied to the target.

The estimator is trained offline on relit tracking frames. We take frames from public tracking videos and apply synthetic, spatially varying illumination: smooth multi-light mixtures in the style of the multi-illuminant scenes in Attentive Illumination Decomposition Model for Multi-Illuminant White Balancing, plus hard cast-shadow masks with colored ambient fill, in the spirit of the shadow-as-local-illuminant view of Conditional GANs for Multi-Illuminant Color Constancy. The ground-truth offset field is known by construction. We additionally pretrain on LSMI and MIIW so that the estimator is not tied to our synthetic lighting.

The analytical part of the program measures how albedo/illumination confusion depends on receptive field. We train matched-parameter variants of the estimator whose receptive fields range from a 7×7 local window, through dilated stacks, to the full criss-cross version. For each variant we measure two quantities: the error of the estimated light field, and how much of the target's own chromaticity is wrongly removed, broken down by the fraction of the window the target occupies. We expect local estimators to strip target color in proportion to how much of the window the target fills, and the long-range estimator not to.

Baselines

The do-nothing reference is TACF with its original Bayes color histogram, run with the authors' settings. TACF reports 62.8% AUC on OTB100 with HOG features at 24 fps. Its deep variant, DeepTACF, reports 66.7% AUC on OTB100 and 0.344 Expected Average Overlap (EAO) on VOT2016 at 23.5 fps. DeepTACF is the strongest competitor in this file on this question, because it is the strongest tracker in the file that already uses a pixel-level color likelihood. We will insert our module into both TACF and DeepTACF. Beating it means a statistically reliable gain over 66.7% AUC on the OTB100 illumination-variation subset, no loss on the background-clutter subset, a higher VOT2016 EAO than 0.344, and at least 20 fps. We found no per-attribute OTB100 illumination-variation number for TACF on Lacuna, so we will rerun TACF ourselves to obtain the subset numbers.

The global color-constancy baseline applies a single illuminant correction to each frame before histogramming. It uses Cross-Camera Convolutional Color Constancy, called C5, which reports 1.77° mean angular error on NUS-8. C5 is the strongest fast global corrector in the file and isolates the question of whether spatial variation matters at all.

The invariant-histogram baselines replace the tracker's colors with normalized rgb, $l_1l_2l_3$, or $m_1m_2m_3$ from Color-based object recognition, or with the bin-ratio similarity of A Robust Tracking System for Low Frame Rate Video.

The local-normalization baseline divides each pixel by a Gaussian-blurred local mean, with the blur width swept. This is the cheapest spatially varying correction available.

The accuracy-ceiling baseline runs AID per frame as a white-balancing preprocessor, whatever its speed. AID reports 2.04° on LSMI and 2.46° on MIIW.

Two ablations isolate the key ingredient. The first is the matched-capacity local-convolution estimator described in the research program. The second removes the temporal background anchoring. For context we also report rStaple, which reports 81.1% distance precision on OTB2015.

All variants share the same training images, the same tracker hyperparameters, and an equal tuning budget, and all are timed on the same hardware. The file's BACF numbers, 61.6% AUC in the TACF paper versus 77.6% "success" in Learning Background-Aware Correlation Filters for Visual Tracking, clearly use different success definitions. We will therefore compare only numbers we recompute under one protocol.

Why not just…?

Why not just white-balance each frame globally? A single correction cannot equalize a target that is half in shadow, and it reacts to scene-wide changes that happen away from the target. Convolutional Color Constancy, C5, and Cascading Convolutional Color Constancy all collapse the image into one chromaticity histogram and discard pixel positions. This alternative may well suffice on OTB sequences where lighting changes globally. If the C5 baseline matches us on the real illumination-variation subsets, we will report that, and we will restrict our claim to the shadow and mixed-light conditions in the relit set.

Why not just use photometric invariants or bin ratios? Gevers and Smeulders show the price of invariance. Invariants that survive changes of light color, namely $m_1m_2m_3$, rely on neighboring pixels sharing surface orientation. The others degrade under changes of light color, and all of them discard lightness cues that separate targets from clutter. The bin-ratio measure is reported to break down beyond 50% clutter or occlusion. The invariant baselines measure this tradeoff directly on the background-clutter subset.

Why not just normalize locally, or enlarge the receptive field with dilated or deeper convolutions? This is the most serious threat, because it is cheap and spatially varying. Our hypothesis is that it removes target color when the target fills the window. Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining reports that deep convolutional stacks have effective receptive fields much smaller than their theoretical ones. The matched-capacity sweep settles this question. If a dilated stack of equal cost matches criss-cross attention, the claim reduces to "large receptive field," not "non-local," and we will say so.

Why not just use deep features, as DeepTACF does? CNN features may already tolerate lighting changes, but the likelihood map in DeepTACF is still a raw color histogram. Testing our module inside DeepTACF shows whether any residual gain remains.

Proposed Experiments

We will run the estimator three ways. First, offline on LSMI and MIIW, measuring angular error against AID. Second, on the relit OTB100 frames, measuring the error of the estimated light field and the amount of target color wrongly removed as a function of target-to-window size. Third, end to end on OTB100 overall and on its illumination-variation and background-clutter subsets (AUC and 20-pixel distance precision), on VOT2016 (EAO, accuracy, robustness), and on a relit OTB100 variant with injected shadows, which exposes failures the original benchmark rarely contains.

The decisive checks are three. The long-range estimator must beat the matched-capacity local estimator by a widening margin as targets grow within the window. Our full tracker must beat C5-corrected TACF on the shadow-relit sequences. The gains must survive at 20 fps or more. If the first check fails, the receptive-field thesis is wrong, regardless of tracking numbers.