EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer

Fei Wang, Dan Guo, Kun Li, Meng Wang

Abstract

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods.

Introduction: The Challenge of Subtle Motion

EulerMormer is the first Transformer-based framework designed for Eulerian Video Motion Magnification (VMM). It acts like a "microscope for time," revealing tiny movements—such as a baby’s breathing or the pulse in a person's neck—that are invisible to the naked eye. By introducing a novel "dynamic filter" that leverages global sparse attention, it effectively amplifies subtle, invisible motions while suppressing the common pitfalls of photon noise and spatial distortion that plague existing CNN-based methods.

Historically, the field has faced two major hurdles:

  1. Photon Noise: Cameras inevitably introduce noise. When you try to amplify a tiny signal (the motion), you often accidentally amplify this noise, resulting in distracting "flickering" artifacts.
  2. Spatial Inconsistency: Forcing magnification can lead to "frequency collapse," where the edges of moving objects become blurry or distorted.
Distinguishing between static and dynamic fields in a video of a sleeping baby to target subtle breathing motion.

While previous deep-learning approaches used Convolutional Neural Networks (CNNs), these are limited by their local receptive fields. They struggle to distinguish between global structure and local noise, often failing to provide the "clean" magnification required for high-fidelity visualization.

Methodology: The EulerMormer Framework

EulerMormer adopts the Eulerian perspective, which treats motion as changes in pixel properties at fixed positions over time, rather than tracking individual particles. The architecture is divided into three distinct phases:

  1. Texture-Shape Disentanglement: Using specialized Transformer encoders, the model separates an image into "texture" (the appearance) and "shape" (the structure). Motion is then defined simply as the difference in shape between frames ($\Delta$shape).
  2. Motion Magnification: The model uses a Dynamic Filter ($\mathcal{F}$) and a Point-wise Magnifier ($M$).
  3. Dynamic Masking Filter (DMF): Instead of standard attention, it uses a "Top-k" sparse strategy. It identifies and keeps only the most important (contributive) features in the attention matrix, effectively "erasing" noise cues. Noise usually lacks the strong, coherent correlation found in structural motion; by only attending to the "Top-k" correlations, the model naturally filters out chaotic signals.
  4. Multi-Scale Gating Regulator (MGR): This dual-path mechanism handles different frequency features to repair motion contours and ensure smooth boundaries.
  5. Amplification Generation: The magnified shape is recombined with the original texture. The Dynamic Filter is applied again here to ensure the final "recoupling" doesn't introduce new artifacts.

To maintain spatial consistency, the authors found that switching to point-wise convolutions ($1 \times 1$) within the magnifier avoids the "checkerboard" artifacts common in standard local convolutions.

Experimental Evaluation

The authors evaluated EulerMormer on both synthetic and real-world benchmarks. Synthetic datasets (Synthetic I, II, III) were used to test magnification accuracy and robustness to Poisson noise and Gaussian blur. Real-world tests included videos of breathing babies, eye movements, and vibrating fabrics.

Comparison of theoretical and realistic magnification results, highlighting EulerMormer's stability and consistency.

Key Results

EulerMormer achieved state-of-the-art (SOTA) results across the board:

  • Quantitative: On the Synthetic-I dataset, it achieved an RMSE of 0.0594 and a PSNR of 25.49 dB, outperforming the previous best method (MDLMM).
  • Robustness: It showed significant performance gains in noisy environments, proving that the dynamic filter effectively handles blur and Poisson noise.
  • Perceptual Quality: On real-world datasets, it scored highest on the MANIQA metric, indicating that the results are not just mathematically accurate but also visually superior to human observers.

Conclusion and Future Work

EulerMormer represents a significant leap forward by being the first Transformer-based framework for Eulerian VMM. By using global context and sparsity as a denoising tool, it effectively separates subtle signals from photon noise.

While the paper demonstrates a significant leap forward, a few areas remain for exploration. Transformers are generally more computationally expensive than lightweight CNNs, and the feasibility of real-time mobile inference remains to be addressed. Additionally, very large "background" motions (camera shake) can still interfere with the extraction of subtle "foreground" motions. Future research may adapt this dynamic filtering strategy for other video restoration tasks, such as deblurring or super-resolution.

Summary updated

Abstract

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods. The source code is available at https://github.com/VUT-HFUT/EulerMormer.
BibTeX
@inproceedings{wang2024eulermormer,
  author = {Fei Wang and Dan Guo and Kun Li and Meng Wang},
  title = {{EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=BaXJRyuUtb},
}
Figure