Optical flow, the per-pixel motion field between video frames, works by matching local brightness patterns. On blank paper, plain cloth, or skin there is no pattern to match, so flow networks either invent motion or report none. The cleanest fix so far is physical: Textureless Deformable Object Tracking With Invisible Markers paints objects with fluorescent dye that shows only under ultraviolet light and records with a synchronized ultraviolet strobe rig. That fix does not carry over to ordinary cameras. The obvious software alternative is to hide a pattern in visible light, the way invisible watermarks hide data in photographs. It runs into a documented contradiction. StegaStamp: Invisible Hyperlinks in Physical Photographs reports that its hidden residual becomes visible noise in smooth regions such as clear sky, which are exactly the regions where motion estimation needs help.
This proposal argues that the contradiction comes from demanding that the hidden signal be readable in a single frame, and that tracking does not need this. We propose to print or display a pattern whose contrast is low enough that, in any one camera frame, it sits near or below sensor noise. It is designed so that a tracker which adds up evidence along a candidate motion path over many frames can recover it reliably. Faint signals are routinely pulled out of overwhelming backgrounds this way in rare-event physics. A detector there accepts an event only when its energy deposits form a continuous track with the expected shape. The new claim is that the aesthetic–functional trade-off for tracking markers has a third axis, the number of frames the tracker integrates. Spending frames lets the pattern drop below human visibility while dense tracking still works on blank deformable surfaces.
Core Claim
Suppose a pattern of per-pixel amplitude $a$ is printed on a surface, observed with sensor noise of standard deviation $\sigma$, and correlated over a patch of $N$ pixels and $T$ frames along the correct motion path. With roughly independent noise, the matching signal-to-noise ratio grows as
$$ \mathrm{SNR}(a, N, T) \approx \frac{a\sqrt{N T}}{\sigma}. $$
Reliable matching needs SNR above some threshold $s^\ast$. The window length therefore has to satisfy
$$ T \gtrsim \frac{1}{N}\left(\frac{s^\ast \sigma}{a}\right)^2, $$
which follows directly from the square-root growth above. Halving the contrast costs four times as many frames. The proposal makes three claims. First, there is a usable operating region where $a$ is below the human detection threshold at normal viewing distance while $T$ is short compared with the time scale of surface deformation. Second, a tracker built around this accumulation gives dense, accurate trajectories on the DOT benchmark from that paper. Third, single-frame flow networks fine-tuned on the same faint pattern cannot do this, because no single frame contains enough evidence.
How This Differs From Prior Work
Prior work on hiding machine-readable signals in images supplies our design tools and sets our constraints. ArtCoder introduced a differentiable simulation of camera sampling and binarization, so decoding errors can steer image synthesis. StegaStamp trains its encoder through a differentiable model of print-and-recapture distortions (perspective warp, blur, color jitter, noise, compression) and reaches 98.7% mean bit accuracy on physical prints. We reuse both ideas to design our pattern. We differ in two ways. Our decoder target is a dense displacement field rather than a bit string. And StegaStamp's smooth-region failure is a constraint we must get around rather than a minor caveat.
AnimateQR sits at the opposite end of the trade-off. It generates animated stylized QR codes with 96% decoding success, and it does so by deliberately adding coarse, visible luminance structure. That is the "make it visible" corner we want to leave.
The fluorescent-marker work is our closest neighbor and our motivation. It shows that markers rescue tracking: on a paper scene, unmarked tracking "failed within 20 frames" while marked tracking survived the whole sequence. It achieves invisibility through chemistry and dedicated hardware. We try to achieve it through contrast and integration time with a standard RGB camera. That paper also documents a residual problem from trigger delays between ultraviolet and visible exposures, which our single-stream approach avoids by construction.
From motion estimation, Learning Accurate Dense Correspondences and When To Trust Them (PDC-Net) models flow as a mixture of reliable matches and outliers. It measures match uniqueness directly from the correlation volume. That lets it detect when a blank region is ambiguous, but all it can do is discard that region. We borrow its uniqueness measure as the functional loss for designing our pattern. We also borrow its idea of calibrated confidence. Our goal is to fill the holes PDC-Net can only flag.
Spring shows how severe these failures are in practice. It reports a 69.6% outlier rate for the best stereo method in sky regions, and outlier rates rising from about 5.7% to over 60% for the best flow method in high-detail regions.
From rare-event physics we borrow a pattern of reasoning, not code. The PandaX-III study Topological background discrimination in the PandaX-III neutrinoless double beta decay experiment separates signal from background by requiring a continuous track with the expected shape, for example charge "blobs" at both ends and a characteristic length. It reaches roughly seven orders of magnitude of background suppression at above 20% signal efficiency. Pulse-shape methods in First Results from the DEAP-3600 Dark Matter Search with Argon at SNOLAB and $\alpha$-event characterization and rejection in point-contact HPGe detectors separate events by matching a known temporal signature. The lesson we import is concrete. When the signal's shape is known in advance, filter for that shape. Accept a detection only when evidence accumulates consistently along a continuous path, rather than when any single measurement looks strong.
Research Program
The work has three parts.
Pattern design. We optimize a single aperiodic, tileable pattern. It must be aperiodic because a repeating pattern would recreate the ambiguity it is meant to cure. Its energy is confined to a spatial-frequency band where human sensitivity is low at the intended viewing distance, while the camera, working closer, still resolves it.
The design objective has two terms, traded off by a weight. The functional term is the correlation-peak uniqueness of the pattern after passing through a StegaStamp-style differentiable capture model, extended with print dot gain, fabric weave, and stretching of the surface. The perceptual term combines a contrast-sensitivity-weighted visibility penalty with an adversarial critic as in StegaStamp. Sweeping the weight traces a family of patterns from clearly visible to sub-threshold.
Background suppression. Scene shading, folds, and any real texture act as background to the embedded signal. Because we know the pattern's frequency band, every frame first passes through a fixed filter matched to that band. The filter removes the slowly varying shading that dominates blank surfaces, and it reduces natural texture outside the band.
Accumulating tracker. On top of the filtered frames runs a tracker that keeps a template for each tracked point, built up along that point's current trajectory. At each new frame, the template is warped by the current deformation estimate and correlated with the filtered frame. The new observation is then folded into the template, and the trajectory is updated. Neighboring points are tied together by a smooth deformation model, so evidence is shared across the surface.
Track acceptance borrows the track-continuity logic from the physics work. A trajectory is trusted only when its accumulated matching score grows roughly as $\sqrt{T}$, as the core equation predicts for a true path. A path locked onto noise does not show that growth. The same growth statistic becomes the per-point confidence we report.
We will implement the tracker by adding a recurrent accumulation state to RAFT-style iterative refinement. That way the comparison with RAFT changes one ingredient rather than the whole architecture.
Analysis. The analysis product is an empirical three-way frontier relating human detectability, tracking error, and integration window. We will also test whether the window scales as $1/a^2$, as the core equation predicts.
Baselines
Naive reference. The do-nothing point is RAFT, pretrained on standard flow data, run on unmodified DOT sequences. The DOT authors report that unmarked tracking failed within 20 frames on a paper scene. Track survival length is therefore our headline metric alongside endpoint error.
Strongest existing method. This is RAFT fine-tuned on DOT, as done in the fluorescent-marker paper. That paper reports that fine-tuning significantly improves accuracy on DOT and on the separate DeSurT dataset, but gives no numbers. No reported result was found on Lacuna for RAFT fine-tuned on DOT, so we will reproduce it. It is the strongest competitor because it uses learned shape priors and real textureless training data, without any embedded signal. Beating it at a pattern contrast that humans cannot detect is the core result. We also run PDC-Net, both as an uncertainty-aware competitor and for its confidence calibration. For context on cross-benchmark strength, it reports a KITTI-2015 endpoint error of 5.22 against RAFT's 5.54 without fine-tuning.
Key ablations. The first isolates multi-frame integration. A single-frame RAFT is fine-tuned on DOT sequences carrying exactly our pattern at the same amplitude. If this matches our tracker, temporal accumulation is not doing the work. The second isolates pattern design. White noise is matched to our pattern in measured visibility. The third removes the band-matched filter.
Upper reference. A clearly visible high-contrast pattern gives the best tracking a pattern can buy when invisibility is ignored.
Fairness. All learned methods get the same training sequences, the same synthetic pattern overlays, the same GPU-hour budget, and the same hyperparameter search budget.
What counts as winning. At a contrast where a forced-choice human detection test is not significantly above chance, our tracker must beat fine-tuned unpatterned RAFT and the single-frame patterned ablation on endpoint error and track survival, and close a substantial part of the gap to the visible-pattern upper reference.
Why not just…?
Why not just print a pattern contrasty enough for single-frame flow and accept slight visibility? This is the real threat. If the curve of visibility against error shows that a barely visible pattern already gives near-ceiling single-frame tracking, the case for integration shrinks to a narrow regime. AnimateQR's need for coarse, high-contrast structure and StegaStamp's smooth-region artifacts both suggest that single-shot readability costs visible contrast. Neither measured this for dense tracking. The frontier experiment settles it directly: we report both curves, single-frame and integrated, on the same visibility axis. The proposal holds only if the integrated curve lies clearly below the single-frame curve, meaning lower error at equal or lower visibility.
Why not just use fluorescent dye and an ultraviolet rig? That approach works and gives excellent ground truth. It needs chemical application, specialized strobe hardware, and correction for misalignment between exposures. It cannot run on a phone, on an existing camera installation, or on materials that cannot be dyed. Our method targets the cases where the surface can be printed or dyed in visible ink but the capture setup is ordinary. We use the dye method as ground truth rather than as a competitor wherever we can access such a rig.
Why not just fine-tune a flow network on textureless data, or add geometric priors? Fine-tuning on DOT helps, but it can only teach the network how blank surfaces tend to deform. It cannot tell the network where a specific point went when the image holds no evidence. Epipolar and planar priors assume rigid scenes, as in Robust Monocular Epipolar Flow Estimation. Face-specific priors such as Facial optical flow estimation via neural non-rigid registration do not transfer to cloth or rope. The fine-tuned baseline above measures how far priors alone go. Our method can also use priors, through its deformation model, so the comparison isolates the added evidence from the pattern.
Why not just run StegaStamp on each frame and decode position codes? StegaStamp's decoder is built to read a 100-bit payload from a rectified region, not to produce a dense displacement field. Its own authors report it is visible in smooth regions. AnimateQR's analysis also indicates that frame-independent encoding flickers and drops out under motion. We include a variant that decodes a StegaStamp-style position payload per frame, at matched visibility, to confirm this empirically.
Proposed Experiments
Synthetic study on DOT. DOT provides about one million frames across roughly 200 sequences of rope, paper, cloth, and hands, with per-frame 3D models and ground-truth trajectories. We attach the candidate pattern to each surface's albedo through the ground-truth template parameterization. Ground truth is used only to render the pattern as if it were physically there; the tracker never sees it. We then pass frames through the capture model. Rope and hands serve as harder, partially textured cases.
At each contrast level we measure endpoint error, the fraction of points within pixel thresholds, track survival length, and the confidence calibration of the growth statistic. Training and testing use disjoint sequences.
Scaling-law check. On the same synthetic data, we measure the shortest window $T$ that reaches a fixed accuracy at each contrast $a$. If the measured slope on log axes is near $-2$, accumulation explains the gain. A flat slope would mean the network is simply exploiting priors.
Physical study. We print sub-threshold patterns on paper and on dye-sublimated cloth. Sequences are captured with consumer phone cameras under room lighting, with sparse manually annotated points and forward-backward consistency as ground truth. Where an ultraviolet rig is available, we add fluorescent markers following the DOT protocol. The unmodified-capture baselines are also evaluated on DeSurT to confirm that our reproduced baselines generalize as the DOT authors report.
Perceptual validation. Human viewers complete forced-choice detection trials comparing patterned and plain prints at specified viewing distances and lighting. The perceptual axis of the frontier is therefore measured, not modeled.
Regression on textured content. On Spring, we check that the band-matched filter and accumulating tracker do not degrade flow on ordinary textured, unpatterned content compared with plain RAFT. A method for blank surfaces should not hurt the rest of the image.
Decisive outcome. The idea is supported if three things hold together. On paper and cloth, at a contrast humans cannot reliably detect, the accumulating tracker outperforms both fine-tuned unpatterned RAFT and single-frame patterned RAFT. The window-versus-contrast slope is near the predicted value. And the required window is short enough for the deformation speeds present in DOT.
The idea is refuted if the single-frame patterned ablation matches it. It is also refuted, at least for these deformation speeds, if the windows that would make the pattern invisible run longer than the time over which the surface deforms appreciably.
References
- Textureless Deformable Object Tracking With Invisible Markers
- StegaStamp: Invisible Hyperlinks in Physical Photographs
- ArtCoder: An End-to-End Method for Generating Scanning-Robust Stylized QR Codes
- AnimateQR: Bridging Aesthetics and Functionality in Dynamic QR Code Generation
- Learning Accurate Dense Correspondences and When To Trust Them
- Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo
- Topological background discrimination in the PandaX-III neutrinoless double beta decay experiment
- First Results from the DEAP-3600 Dark Matter Search with Argon at SNOLAB
- $$\alpha $$-event characterization and rejection in point-contact HPGe detectors
- Robust Monocular Epipolar Flow Estimation
- Facial optical flow estimation via neural non-rigid registration