Two-Key Correspondence: Chroma-Gated Label Propagation for Self-Supervised Video Tracking

Self-supervised video trackers learn to match pixels across frames from unlabeled video, then use those matches to carry an object mask from the first frame to the rest. Most of them see the world mainly through brightness. The colorization-based tracker removes color from its input by design. Correspondence learners trained on RGB tend to discount color as well, because brightness edges explain most frame-to-frame matches and because heavy color augmentation teaches them to ignore hue. This creates a blind spot. When an object and its background differ in hue but not in brightness, which is called an isoluminant boundary, or when a nearby distractor has the same texture but a different color, the matching spreads the mask label across the boundary and the track drifts.

We propose to train two separate matching signals from unlabeled video. One sees only luminance and the other sees only intensity-normalized chromaticity. Each is forced to be useful on its own by having it reconstruct the channel it cannot see. At propagation time a reference pixel may pass its label to a target pixel only if both signals agree. The chroma signal acts as a spatial attention mask on the luminance-driven propagation, the same way layout masks gate attention in diffusion models. The claim we test is narrow. If chroma is learned as its own correspondence signal and used to gate propagation, rather than mixed into one RGB embedding, mask leakage across low-luminance-contrast boundaries should drop, and overall accuracy on DAVIS 2017 should be no worse.

Why this, and what is already known

Three bodies of work motivate the gap. Tracking Emerges by Colorizing Videos trains a pointer from a grayscale target frame to a color reference frame. Its copy constraint forces it to reconstruct color as a weighted average of reference colors. This yields tracking that beats FlowNet2 on DAVIS 2017 (J of 34.6 versus 26.7), but its embedding sees luminance only, so isoluminant structure is invisible to it by construction. Learning Correspondence from the Cycle-Consistency of Time and Space-Time Correspondence as a Contrastive Random Walk replace colorization with forward-backward consistency. The second, CRW, treats patches as nodes in a space-time graph and trains the model so that a random walk out and back along a palindromic clip returns to its start, reaching 67.6% J&F on DAVIS 2017. Neither paper makes color a separately supervised signal. On the color side, Color Descriptors for Object Category Recognition shows that opponent and normalized-chromaticity descriptors add a 30% relative gain over intensity-only features. Efficient Color Boundary Detection with Color-Opponent Mechanisms shows that color-opponent channels recover boundaries luminance misses (F of 0.64 on BSDS300). These results tell us the missing information exists and is cheap to compute. They do not show how to learn it from video.

The gating mechanism comes from two places. Packing and Padding: Coupled Multi-index for Accurate Image Retrieval suppresses false matches by requiring that a candidate co-occur in both SIFT-word space and color-name space. Our gate is a learned, dense, temporal version of that "two keys must match" rule. Freestyle Layout-to-Image Synthesis sets out-of-mask attention scores to negative infinity. Dense Text-to-Image Generation with Attention Modulation argues instead for soft modulation that preserves global context. This hard-versus-soft tension carries over to our setting, and we test both.

The closest systems that combine masking with temporal coherence take their masks from outside. ConsisDrive masks video-diffusion attention with ground-truth 3D boxes. CustomVideo uses masks from a pretrained segmenter. Our mask is produced by the model's own self-supervised chroma correspondence, and it acts on label propagation rather than generation. Learned Variational Video Color Propagation fuses flow and VGG matches to propagate color. It uses fixed features and outputs color. We learn the features, and color is used as a constraint on propagating labels.

One constraint from the color literature shapes the design. Why You Should Forget Luminance Conversion and Do Something Better shows that standard grayscale conversions err by 30% to over 50% relative to true luminance because they ignore gamma and camera tone curves. For a colorization pretext, this means the "luminance" input may still carry chroma, which offers the network a shortcut. We treat conversion choice as an experimental variable rather than a detail.

Core Claim

The claim has three parts. First, a correspondence signal computed from chromaticity alone can be learned from unlabeled video if it is trained to copy luminance. Second, its information is not recovered by an RGB correspondence learner trained the usual way. Third, gating luminance-based propagation with it reduces label leakage at low-luminance-contrast boundaries.

Concretely, let $s^L_{ij}$ and $s^C_{ij}$ be the cosine similarities between target pixel $i$ and reference pixel $j$ under the luminance encoder and the chroma encoder. Propagation weights are

$$A_{ij} = \mathrm{softmax}_j\!\left(\frac{s^L_{ij}}{\tau} + \lambda \log \sigma\!\left(\frac{s^C_{ij}-\theta}{\kappa}\right)\right),$$

where $\tau$ is a temperature, $\sigma$ is the logistic function, $\theta$ is a chroma-agreement threshold, $\kappa$ sets the gate's sharpness, and $\lambda$ sets its strength. As $\kappa \to 0$ the gate becomes the hard negative-infinity mask of FreestyleNet. With a large $\kappa$ it becomes a soft modulation in the spirit of DenseDiffusion.

Research Program

We build two ResNet-18 encoders with matching capacity. The luminance encoder takes linearized luminance. The chroma encoder takes intensity-normalized rg chromaticity, the representation behind rgSIFT, which is invariant to shading and intensity changes. That invariance addresses the tension the file notes: raw color is unstable under lighting changes across frames, but chromaticity is much less so.

Both encoders are trained on Kinetics-400 with three losses. The first is the CRW palindrome walk loss computed on the gated affinity above, so the gate is trained end to end. The second is a Vondrick-style copy loss: the luminance encoder's affinity must reconstruct the target frame's chroma from the reference frame. The third is the mirror-image copy loss: the chroma encoder's affinity must reconstruct the target frame's luminance from the reference. The mirror loss is the new ingredient. Without it, the chroma encoder can coast on the walk loss whenever the luminance stream already solves the walk, and it would learn nothing about the isoluminant cases.

The analysis has three threads. The first measures how often the two affinities disagree, and where, against ground-truth boundaries. The second checks whether the gate's effect concentrates on boundaries with low luminance contrast, as the hypothesis predicts. The third compares a standard sRGB-weight luma input against a gamma-linearized one, to test whether conversion error lets the colorization pretext cheat.

Baselines

The naive reference point is label propagation using raw-pixel affinity in Lab space and, separately, ImageNet-supervised ResNet-18 features. Both use the same propagation procedure: the same number of context frames, the same top-k, and the same local radius. This tells us how much any learning adds.

The historical anchor is the colorization tracker of Tracking Emerges by Colorizing Videos, which reports J of 34.6 on DAVIS 2017. Learning Correspondence from the Cycle-Consistency of Time reports J of 41.9 and JHMDB [email protected] of 78.5%.

The strongest competitor is CRW from Space-Time Correspondence as a Contrastive Random Walk, which reports DAVIS 2017 J&F of 67.6%, or 68.3% with test-time adaptation. It is the strongest because it already sees RGB and already uses temporal cycle supervision, so any gain we show must come from how chroma is learned and used, not from merely having it. We will retrain CRW ourselves on the same Kinetics-400 data, with a matched parameter count and the same training schedule and propagation settings. We give it the same hyperparameter search budget as our method. We report it against both our reproduction and the published 67.6% figure. Supervised context comes from The 2017 DAVIS Challenge on Video Object Segmentation, where the top team reached 69.9% J&F.

Beating CRW means two things together. Our method must show a clear gain on the low-luminance-contrast stratum, larger than the spread across three training seeds of either method. Its overall J&F must also be within seed noise of our CRW reproduction or above it. A gain only on the stratum, with an overall loss, would count as a partial result, not a win.

Four ablations isolate the key ingredients. Removing the gate, by summing the two affinities, tests whether gating matters beyond feature fusion. Removing the mirror copy loss tests whether the chroma stream needs its own supervision. Comparing hard and soft gates resolves the FreestyleNet-versus-DenseDiffusion tension in this setting. A single RGB encoder trained with the walk loss plus colorization tests whether splitting the input into two streams matters at all.

Why not just…?

Why not just train CRW on RGB with weak color augmentation? This is the most serious threat. An RGB encoder in principle has all the chroma information. If CRW trained with color jitter reduced or removed matches our method on the low-contrast stratum, the two-stream design is unnecessary and we would report that. Our hypothesis is that the walk loss is mostly satisfied by luminance structure in Kinetics, so an RGB encoder has little pressure to encode isoluminant edges. Only the mirror copy loss creates that pressure. We run this baseline at several jitter strengths with the same compute budget. The probe of the chroma encoder, described under the decisive checks, tells us directly whether RGB-CRW features already encode what the chroma encoder learns.

Why not just gate with a fixed color feature such as Lab chroma distance or color names, as in coupled multi-indexing? Packing and Padding shows that fixed color names can suppress false matches in retrieval. The same paper's evidence, together with the heuristic tuning noted for opponent descriptors, suggests that fixed color features are brittle under the illumination changes common in video. We run a fixed-gate variant, using rg-chromaticity distance and color-name agreement, inside the same propagation code. If it matches the learned gate, the contribution shrinks to "gate propagation with color," which is still useful but simpler, and we would say so.

Why not just restrict propagation to a small local window or a small top-k? Locality already limits long-range leakage, and every propagation baseline here uses it. It cannot separate an adjacent same-brightness region from the true object, which is exactly the failure we target. The stratified evaluation shows whether locality alone closes the gap.

Why not just use optical flow? Tracking Emerges by Colorizing Videos reports FlowNet2 propagation at J of 26.7, well below learned correspondence. Learned Variational Video Color Propagation needed heavy refinement on top of RAFT to handle occlusion, so we include flow only as a reference.

Proposed Experiments

We evaluate on DAVIS 2017 validation, reporting J&F, J, and F, and on JHMDB keypoint propagation, reporting [email protected] and [email protected].

The decisive check is a stratified boundary analysis on DAVIS 2017. For each ground-truth object boundary we measure the luminance contrast and the chromaticity contrast across it, using linearized luminance. We then report boundary F and leakage separately for three groups: low-luminance-contrast boundaries with high chroma contrast, low-contrast boundaries with low chroma contrast, and high-contrast boundaries. The hypothesis predicts gains concentrated in the first group and no gain in the second, which serves as a control.

A second check is a controlled stress set. We build it from DAVIS sequences by remapping each foreground object's luminance to match its local background while keeping its chromaticity, which turns real boundaries into isoluminant ones. If our method does not degrade less than CRW on this set, the claim fails.

A third check tests whether the chroma encoder learns something RGB-CRW lacks. We freeze each encoder and fit linear probes that predict the chroma-encoder features from the RGB-CRW features. If the RGB-CRW features explain the chroma features almost entirely, the information is already there, and any gain must come from the gate alone.

The conversion experiment trains the luminance stream on standard sRGB-weight luma and on gamma-linearized luminance. We then measure how well a probe can recover chroma from each luminance input. This tests the shortcut risk raised by Nguyen and Brown.

All results use three seeds.