Borrowing Phase Evidence Across an Object to Unwrap Large Fine-Scale Motion

Phase-based video models describe motion as a change in local phase, meaning the position of a small oscillating wave pattern inside each patch of the image. Sliding an edge by a small amount shifts that phase by a proportional amount. This makes motion close to linear and lets very small networks predict and interpolate video. The weakness is that phase repeats every $2\pi$. A fine-scale pattern that moves more than half its wavelength looks the same as one that moved a little, or one that moved a little in the opposite direction. Today's systems resolve this ambiguity at each location by consulting a blurrier, coarser copy of the same spot. That fails for thin, fast, high-frequency structures such as spokes, cables, strokes, and text, because they have little or no energy at coarse scales.

We propose to resolve the ambiguity across space instead of across scale. Each fine-scale patch is allowed to collect phase evidence from distant patches that belong to the same moving object, using a cheap global routing step. A thin antenna can then be unwrapped using evidence from the car body it is attached to, without ever downsampling the antenna. The main contribution is a wrap-aware way to pool that evidence: patches exchange periodic likelihoods over velocity, not raw phase numbers. The idea can be tested cleanly in a regime where coarse-to-fine methods are predicted to fail.

Core Claim

Coarse-to-fine unwrapping borrows disambiguating evidence from the same location at a coarser scale. Non-local unwrapping borrows it from the same object at any scale. When a structure's coarse-scale energy is weak but its object has strong coarse or multi-orientation energy elsewhere, non-local unwrapping should recover the correct displacement where coarse-to-fine cannot.

This only works if the pooling respects phase periodicity. Averaging wrapped phase values with ordinary attention destroys the information. Averaging complex per-band phase-difference coefficients does not. The claim is falsifiable. If a coarse-to-fine unwrapping baseline matches our method on thin, fast structures, the idea is wrong.

How the Mechanism Works

The input is a local frequency decomposition of two consecutive frames. Following Local Frequency Domain Transformer Networks for Video Prediction, this is a local Fourier transform over overlapping cells. A complex steerable pyramid as in PhaseNet for Video Frame Interpolation also works.

For each cell $x$ and each frequency band $b$, define three quantities:

  • $\boldsymbol\omega_b$ is the band's spatial frequency vector.
  • $A_b(x)$ is the band's amplitude.
  • $\Delta\phi_b(x)$ is its wrapped phase change between frames.

The phase change constrains the cell's velocity $\mathbf v$ only up to whole periods. A natural wrap-aware score for each candidate velocity is a von Mises (circular Gaussian) log-likelihood:

$$\ell_x(\mathbf v)=\sum_b A_b(x)\,\cos\!\big(\boldsymbol\omega_b\cdot\mathbf v-\Delta\phi_b(x)\big).$$

For a single fine band this score is a periodic comb of equally good candidates. The comb is the phase-wrapping ambiguity made explicit.

The score is a Fourier series in $\mathbf v$ whose coefficients are the complex numbers $c_b(x)=A_b(x)e^{i\Delta\phi_b(x)}$. Pooling likelihoods across cells is therefore exactly linear pooling of these complex coefficients. We aggregate them with learned affinities:

$$L_x(\mathbf v)=\ell_x(\mathbf v)+\sum_{y}a_{xy}\,\ell_y(\mathbf v).$$

The affinities $a_{xy}$ come from recurrent criss-cross attention, the two-pass row-then-column routing of CCNet: Criss-Cross Attention for Semantic Segmentation. They are computed on amplitude features only, the "what" channel, so that grouping depends on appearance while the "where" evidence stays in the phase coefficients.

The velocity estimate is a soft-argmax of $L_x$ over a velocity grid. Its range can far exceed the finest wavelength, because once the right cells are pooled, bands with different frequencies and orientations break the comb symmetry. Each band's phase is then advanced by $\boldsymbol\omega_b\cdot\mathbf v$ and inverted to produce the next frame or an intermediate frame.

A small convolutional correction handles disocclusion, as in Motion Segmentation using Frequency Domain Transformer Networks. The whole system is trained end to end on frame reconstruction. Classical phase correlation over a whole image is the special case where every affinity equals 1. That case fails on multi-object scenes. Our method can be read as phase correlation over learned, object-shaped supports.

Relation to Prior Work

The motivation comes from the phase literature.

None of these routes phase evidence between distant locations. That routing is the one thing we add.

The routing method is reused from CCNet, which reports 11x less memory than dense non-local blocks. It is chosen over Compact Generalized Non-local Network for cost. We apply it to complex likelihood coefficients, not to semantic features.

Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening constrains the design. It argues that spatial operators are invalid on a global spectrum because frequency bins are not translation invariant. Our attention runs over spatial cells, where translation invariance holds, while the frequency axis is handled analytically through $\boldsymbol\omega_b$ and never convolved. This is how we avoid the mismatch it identifies.

Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture uses sparse transformers for long-range context in motion magnification, but without phase algebra. It is the closest "global context for motion" design, and it differs in exactly the ingredient we test.

Research Program

The work has three stages.

The first stage is a diagnostic benchmark that isolates the regime where coarse-to-fine unwrapping should break. We build composite synthetic objects with a large, low-frequency body rigidly attached to thin one-to-two-pixel strokes. Per-frame displacements are swept from 0.2 to 3 times the finest-band wavelength. A separate knob is the fraction of each object's energy that lives only at fine scales. These sequences are rendered over STL-10 backgrounds following the Moving MNIST on STL protocol of FDTN, and as multi-object Moving MNIST++ variants following LFDTN. Ground-truth velocity is known for every cell, so we can measure a wrap-error rate: the fraction of cells assigned the wrong integer number of periods.

The second stage builds and ablates the model described above in an LFDTN-style video prediction setting, keeping the parameter count in the low thousands to tens of thousands.

The third stage transfers the same unwrapping module into a PhaseNet-style interpolator. That interpolator is evaluated on Middlebury and Adobe240 to check that the gain survives real footage. On NGSIM traffic footage, LFDTN reported that its velocity maps are interpretable, and we check whether ours stay coherent on thin vehicle features.

Baselines

The naive reference is copy-last-frame for prediction and frame averaging for interpolation. These bound how much any motion model helps.

The strongest existing phase method on our main benchmark is LFDTN. It reports L1 error 0.00589 on Moving MNIST++ with about 3,000 parameters, against 0.00681 for the 15.8M-parameter HPNET. On Moving MNIST over STL-10 backgrounds, FDTN reports MSE 0.0002 and SSIM 0.9896 with 2,000 parameters. We rerun both from their published configurations on the standard splits to confirm these numbers, then on our diagnostic sweeps. Beating LFDTN means matching its standard Moving MNIST++ L1 within noise and lowering wrap-error rate and L1 substantially once displacement exceeds half the finest wavelength.

The most dangerous competitor is coarse-to-fine unwrapping. We build it ourselves because it is the mechanism prior phase networks rely on implicitly. It is LFDTN with a pyramid in which each fine level's period count is chosen from the upsampled coarse-level estimate, with the same parameter budget as our model. Beating it means a clear wrap-error reduction specifically when fine-only energy is high. Parity there falsifies the core claim.

Two ablations isolate the key ingredient. The first applies identical criss-cross routing to raw wrapped phase values treated as real numbers, testing whether wrap-aware pooling matters. The second sets all affinities to 1, giving global phase correlation, which tests whether learned grouping matters.

A pixel-space comparison uses the same criss-cross routing on raw frames predicting a flow field that warps the last frame. It answers whether phase is needed at all.

All models get the same training data, the same number of gradient steps, and the same hyperparameter search budget of twenty trials per method. Parameter counts are reported alongside every result.

For Adobe240, V³ reports more than 2 dB PSNR over VideoINR, but the file gives no absolute value. For PhaseNet on Middlebury, no reported result was found on Lacuna. For both benchmarks we therefore rerun the original PhaseNet as the reference rather than quote a number.

Why not just…?

Why not just rely on the multi-scale pyramid, as PhaseNet already does? This is the real threat. For most natural content, coarse levels probably unwrap fine phase correctly, and our method may add nothing there. The argument against it is structural. A one-pixel stroke contributes almost nothing to coarse bands at its own location, so a same-location coarse estimate is noise. PhaseNet's reported softness on fine detail is consistent with this, though not proof. The fine-only-energy sweep settles the question directly. If coarse-to-fine matches us at high fine-only fraction, we drop the claim. If it only loses there, the contribution is real but regime-specific, and we will say so.

Why not just put a transformer or non-local block on raw pixels? Dense attention costs grow with the square of the pixel count. ABFNet also argues that self-attention behaves like a low-pass filter, which is the wrong bias for thin, fast structures. The pixel-space criss-cross baseline uses the same routing and the same compute. If it matches our method at equal parameters, then phase is not the active ingredient and the routing alone explains the result.

Why not just widen the receptive field with large or dilated kernels on the phase sub-bands? Kernels mix wrapped phase values linearly, which is exactly the operation our raw-phase ablation shows to be harmful. Dilated sampling also skips the fine structures we care about, and FD4MM shows artifacts on abrupt motion despite using dilated smoothing. We include a dilated-kernel variant that operates on the complex coefficients $c_b$, with receptive field matched to two criss-cross passes. It differs from our model only in having fixed rather than appearance-dependent pooling.

Why not just use a global FFT, which already has a full receptive field? A single global spectrum assumes one coherent shift. With several independently moving objects, the phase contributions interfere, which is why FDTN and LFDTN window locally. Our all-ones affinity ablation is exactly this global phase correlation. We expect it to work for a single object and to collapse on multi-object Moving MNIST++.

Decisive Checks

The central experiment plots wrap-error rate and prediction L1 against two axes: displacement measured in finest-band wavelengths, and the fraction of an object's energy that exists only at fine scales.

The prediction is specific. Coarse-to-fine and our method should agree when fine-only energy is low. Coarse-to-fine should degrade sharply as fine-only energy rises. Our method should stay accurate as long as some part of the same object carries coarse or multi-orientation energy.

Two negative controls bound the claim. First, an object made purely of fine texture with no coarse energy anywhere should defeat both methods, and our method should not invent confident wrong velocities. Second, adjacent objects with identical appearance moving in different directions probe whether appearance-based grouping leaks evidence across object boundaries. That failure would show up as a correlated wrap error along shared borders.

On real data, the claim is supported if three things hold together:

  • We match the standard Moving MNIST++ and STL results.
  • We keep PhaseNet-level PSNR on Middlebury and Adobe240.
  • We gain specifically on high-motion crops containing thin structures.

A uniform small gain everywhere would instead suggest that the extra routing capacity, not unwrapping, is doing the work.