Video motion magnification makes invisible motion visible. It takes a video of a vibrating bridge, a breathing infant, or a pulsing wrist and amplifies sub-pixel displacements until a person can see them. The dominant failure is the artifact. Camera sensor noise gets amplified into shimmering ripples in regions that should be still, and moving edges blur or tear. Learned magnifiers built from small convolutional filters cannot tell the two apart well, because they look at a few pixels at a time. Within a 3×3 window, a flicker of noise and a genuine 0.05-pixel shift look much the same. EulerMormer showed that giving the network a global view, through sparse attention across the whole feature map, suppresses this noise. That approach pays the cost of attention on high-resolution video, and it treats the global view as a generic learned operator.
This proposal starts from a simpler physical fact. When a patch of image translates by a small vector $d$, its Fourier transform is multiplied by a phase ramp, and its magnitude is unchanged. Real motion therefore produces a phase relationship between two frames that is consistent across many feature channels at each frequency. Noise produces random phase. We propose a magnifier whose global operator is a spectral coherence gate. It takes the Fourier transform of the network's motion features over large windows, measures at each frequency how consistently the two frames agree across channels, and keeps only coherent frequencies before amplification. The operator costs $O(N\log N)$ in the number of pixels $N$. It has no built-in low-pass bias, so it can keep sharp edges. Because frequency orientation corresponds to motion direction, it can also be made directional at almost no extra cost.
Core Claim
The claim is that the noise-versus-motion ambiguity in learned Eulerian magnification is mainly a problem of coherence estimation, not of generic long-range context. An explicit coherence measurement in the frequency domain should remove noise-induced flicker as well as global attention does, at the runtime of a lightweight convolutional network. It should also avoid the edge smoothing that attention-style mixing is known to cause.
To make the mechanism concrete, let $F_0^c(\omega)$ and $F_t^c(\omega)$ be the windowed Fourier transforms of channel $c$ of the "shape" features of the reference frame and the target frame, at spatial frequency $\omega$. Here "shape" means the structural branch of the texture/shape encoder introduced by Learning-Based Video Motion Magnification. We compute the magnitude-squared coherence across channels:
$$\gamma(\omega)=\frac{\left|\sum_c F_0^c(\omega)\,\overline{F_t^c(\omega)}\right|^2}{\sum_c |F_0^c(\omega)|^2\;\sum_c |F_t^c(\omega)|^2}.$$
For a pure local translation, $F_t^c=F_0^c e^{-i\omega\cdot d}$, so $\gamma=1$ whatever the size of $d$. For independent noise, $\gamma$ falls toward roughly $1/C$, where $C$ is the number of channels. This follows because the cross terms add with random phase. The actual gain on learned features, whose channels are correlated rather than independent, is an empirical question the ablations must answer.
The gate applied to the spectrum of the feature difference is
$$g(\omega)=\sigma\big(a\,\gamma(\omega)+b_\phi(\omega)\big).$$
Here $\sigma$ is the sigmoid, $a$ is a learned scale, and $b_\phi$ is an optional learnable anisotropic Butterworth-style term. That term follows Butterworth as Attention and is oriented by a user-chosen motion axis $\phi$. The gated difference is transformed back and passed to a point-wise $1\times1$ magnifier, reusing EulerMormer's finding that spatial kernels at the scaling step create checkerboard artifacts.
Relation to Prior Work
Four papers motivate the proposal. EulerMormer is the closest work: it diagnoses the local receptive field as the cause of flicker and blur, and fixes it with top-$k$ sparse cross-covariance attention. We keep its diagnosis and its point-wise magnifier. We replace its learned attention with an explicit, cheap coherence statistic, and we test directly whether the attention was doing anything beyond estimating coherence.
Lightweight Network for Video Motion Magnification builds a real-time purely convolutional model. It reports about 7.6× fewer parameters than Oh et al., and 38 ms versus 95 ms at 720×720. Its authors acknowledge edge smoothing and a gap to the larger model. That speed is our budget target.
Learning-based Axial Motion Magnification separates motion along a chosen axis with a convolutional projection module, but it has no global view. Our $b_\phi$ term instead offers directionality inside the global operator.
Butterworth as Attention supplies the method for learnable anisotropic spectral gating and the argument that transformers act as low-pass filters. It gates a single image's spectrum. It does not compare two frames, and it was never applied to motion.
Two papers constrain the design. The classical phase-correlation idea behind discontinuous phase correlation for ice floes reminds us that a single global Fourier transform mixes objects moving differently and lowers their coherence. For that reason the gate works on overlapping windows, and window size is a studied variable, not a fixed choice. Oh et al. warns that hand-crafted spectral filters ring at high-contrast edges. Our gate is learned end to end and sits in feature space rather than pixel space, but ringing is still a named failure we measure.
Several receptive-field papers supply alternative global operators that we use as controls, not as the contribution. These are criss-cross attention in CCNet, cross-channel non-local blocks in CGNL, and oversized 1D kernels in ParCNetV2.
Research Program
We will build the magnifier on the Singh et al. lightweight encoder–decoder, keeping its appearance encoder and decoder. The coherence gate is inserted between the shape-difference computation and a $1\times1$ magnifier, with a parallel 7×7 depth-wise local path in the style of ParCNetV2's local-global block, so that small isolated motions are not lost when windows mix. Training uses the standard synthetic data of Oh et al.: PASCAL VOC foregrounds on MS COCO backgrounds, sub-pixel shifts from 0.01 to 10 pixels, Poisson noise, and quantization. For the axial variant we add the axial copy-paste data of Learning-based Axial Motion Magnification.
The analysis has three parts. First, we measure $\gamma(\omega)$ on held-out synthetic pairs with known motion masks, and check whether learned shape channels actually give the predicted gap between coherence in moving regions and in noise-only regions. If they do not, the premise fails before any benchmark is run. Second, we sweep the window size from 16 to the full frame to map how coherence is lost when several objects move differently in one window. Third, we extend the cross-spectrum sum from two frames to a short temporal stack. This addresses the two-frame limitation shared by EulerMormer and FlowMag, and tests whether temporal averaging of the coherence estimate helps at magnification factors above 20×, where Oh et al. report breakdown.
Baselines
The naive reference point is the Singh et al. lightweight network retrained unchanged on our data. It is what the gate is added to, so any improvement must appear against it at nearly the same runtime. Its reported efficiency is 38 ms at 720×720. For its reconstruction accuracy, the file notes that it achieves the lowest mean absolute error and mean squared error among the methods it compares, but gives no values. We therefore report no reported result found on Lacuna for its synthetic SSIM or MAE, and we will re-measure it ourselves.
The strongest existing method on the artifact question is EulerMormer, because it is the only prior work that uses a global receptive field specifically to separate noise from motion. We will run the authors' model where released, or a faithful reimplementation otherwise, on Synthetic I–III and our noise sweep, with identical training data and the same number of training iterations. No reported result was found on Lacuna for EulerMormer's SSIM or PSNR values on those sets, so all numbers will be produced in our own harness. Beating it means matching or exceeding its SSIM at every Poisson noise level while running at least three times faster at 720×720 on the same GPU. A tie on quality at that speed would still support the claim. Losing on quality at high noise would refute it.
For the directional setting, the baseline is Learning-based Axial Motion Magnification, evaluated on its Generic and Axial Evaluation Datasets under its own protocol. The file reports it outperforming phase-based methods in SSIM but gives no values, so we will re-measure. For self-supervised real-video motion error, FlowMag's protocol across 2× to 64× is the reference. The file reports lower motion error than supervised baselines across that range but gives no values, so FlowMag's number will also be re-measured.
Two ablations isolate the key ingredient, and each is given the same parameter count and the same training budget. The first keeps the Fourier transform but removes $\gamma$, leaving only a learned static Butterworth gate. This separates "global spectral mixing" from "coherence measurement." The second replaces the gate with recurrent criss-cross attention or a CGNL block at matched FLOPs. This separates "coherence" from "any efficient global operator."
Why not just…?
Why not just denoise the frames first with a strong off-the-shelf denoiser and then run the lightweight magnifier? This is the most serious threat, because flicker is amplified noise. The difficulty is that a single-frame denoiser removes exactly the sub-pixel intensity changes that encode 0.01-pixel motion, which should shrink the magnified displacement. The file contains no reported result on denoise-then-magnify, so we will test it directly. We will run a denoiser pre-pass before the Singh et al. network and measure both background flicker and magnified-displacement accuracy against synthetic ground truth. If this pipeline matches the coherence gate on both at comparable runtime, the proposal's contribution reduces to engineering, and we will say so.
Why not just add criss-cross attention or another efficient non-local block to the lightweight network? This is the obvious move, listed as an open gap in the file. It is exactly our second ablation. CCNet reports 11× less memory and 85% fewer FLOPs than dense non-local blocks, so it is a fair efficient competitor. What it lacks is any reason to prefer phase-consistent evidence over appearance similarity. If it matches the gate, the coherence hypothesis is not what matters, and a generic global view suffices.
Why not just use classical temporal filtering over many frames, which already exploits the fact that noise is temporally incoherent? Classical Eulerian pipelines do this. Oh et al. show that their hand-crafted spatial decompositions ring at edges, and Cai et al. show that fixed temporal bands fail on non-stationary vibrations. We include phase-based magnification and BE-SCT on the real vibration sequences. Our multi-frame coherence extension also tests whether temporal evidence and spatial-spectral evidence are complementary rather than redundant.
Why not just use dilated or deeper convolutions? EulerMormer reports that deeper local hierarchies still blur boundaries and cause frequency collapse. We include a dilated-convolution variant at matched FLOPs as a sanity row, rather than argue the point.
Proposed Experiments
On synthetic data, we will report SSIM, PSNR, and MAE on the Oh et al. test set and Synthetic I–III, sweeping Poisson noise level and magnification factor up to 64×. On real videos, we will use the baby-breathing, cat-toy, water-bottle, and hammer sequences, together with FlowMag's 650 real test pairs. Here we report FlowMag-style motion error. We also report a static-region flicker score: the temporal standard deviation of output intensity inside hand-annotated still background regions, measured on real sequences. For directional control, we report leakage on the Axial Evaluation Dataset, defined as the energy of magnified motion orthogonal to $\phi$. Runtime and memory are reported at 720×720 and 1080p.
Two checks decide the outcome. The first is the coherence-versus-static-gate ablation: if removing $\gamma$ costs little, the physical story is wrong. The second is the denoise-then-magnify pipeline: if it matches the gate on both flicker and displacement accuracy, the method is unnecessary.
A result showing the gate matching EulerMormer's noise robustness near Singh et al.'s runtime, with lower edge blur than criss-cross attention at matched cost, would establish that frequency-domain coherence is the right global operator for magnification. We also expect a specific failure: coherence should drop in windows containing opposing motions. The window-size sweep will measure how often this occurs on real footage and whether the local path compensates for it.
References
- EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer
- Learning-Based Video Motion Magnification
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- Lightweight Network for Video Motion Magnification
- Learning-based Axial Motion Magnification
- Discontinuous Non-Rigid Motion Analysis of Sea Ice using C-Band Synthetic Aperture Radar Satellite Imagery
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Compact Generalized Non-local Network
- ParCNetV2: Oversized Kernel with Enhanced Attention