EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer

Fei Wang, Dan Guo, Kun Li, Meng Wang

Abstract

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods.

TL;DR

EulerMormer is the first Transformer-based framework designed for Video Motion Magnification (VMM). By combining Eulerian motion theory with a novel dynamic filtering strategy, it effectively amplifies imperceptible movements while suppressing the "flickering" artifacts and motion blur that plague traditional CNN-based methods.

The Challenge of Motion Magnification

Video Motion Magnification (VMM) acts like a "microscope for time," revealing tiny movements—such as a pulse in a neck or the vibration of a mechanical part—that are invisible to the naked eye. However, two major hurdles have limited its real-world use:

  1. Photon Noise: Cameras inevitably introduce noise. When you amplify tiny motions, you often accidentally amplify this noise, leading to distracting flickering in static areas of the video.
  2. Spatial Inconsistency: Forced magnification can cause "frequency collapse," resulting in motion blur, ringing artifacts, and distorted edges in the dynamic parts of the frame.
Theoretical magnification and the challenge of flickering artifacts in static fields.

Existing methods mostly use Convolutional Neural Networks (CNNs). While effective, CNNs have a limited "receptive field" (they only look at local clusters of pixels), making it difficult to maintain global spatial consistency during the magnification process. As shown in the comparison above, traditional methods often suffer from attenuation and flickering, whereas EulerMormer maintains stability and consistency across both static and dynamic fields.

The EulerMormer Architecture

The authors introduce EulerMormer, which shifts the architecture from local CNNs to global Transformers. The pipeline consists of three main phases designed to handle motion and noise separately.

The EulerMormer framework architecture, highlighting the three-phase pipeline and dynamic filtering components.

Phase 1: Texture-Shape Disentanglement

Based on Eulerian theory, the model separates the video into "Texture" (the appearance) and "Shape" (the geometry). Motion is then represented as the difference in shape between frames ($\Delta Shape$). This disentanglement allows the model to process motion signals without degrading the fine details of the object's appearance.

Phase 2: Motion Magnification with Dynamic Filtering

This is the core innovation of the paper. It uses a Dynamic Filter ($F$) composed of two specialized modules:

  • Dynamic Masking Filter (DMF): This module employs a "Top-k" sparse cross-covariance attention mechanism. By only attending to the most relevant features, it explicitly prunes out low-correlation noise cues, effectively treating denoising as a selection problem.
  • Multi-Scale Gating Regulator (MGR): A dual-path network that uses different kernel sizes (1x1, 3x3, 5x5) to capture motion at various frequencies. This helps "repair" and smooth motion contours that might otherwise be lost during magnification.

Phase 3: Amplification Generation

The magnified shape is recombined with the original texture. A Point-wise Magnifier (PWM) is used here instead of standard convolutions to prevent checkerboard artifacts and ensure global consistency across the entire frame.

Experimental Results

EulerMormer was evaluated against state-of-the-art (SOTA) methods like MDLMM and LNVMM across synthetic and real-world datasets.

  • Quantitative Performance: On the "clean" Synthetic-I dataset, EulerMormer achieved a PSNR of 25.49 dB (outperforming the previous best of 24.84 dB) and the lowest magnification error (RMSE of 0.0594).
  • Robustness to Noise: In scenarios with Poisson noise (Synthetic-II), the model maintained a high PSNR of 25.04 dB, while competing methods saw significant performance drops. This demonstrates the effectiveness of the DMF's sparse attention in filtering out noise.
  • Real-World Quality: Using the MANIQA metric (a no-reference image quality assessment), EulerMormer scored highest across all real-world categories, including static scenes (e.g., a breathing baby) and dynamic scenes with background movement. The results indicate more realistic and stable visualizations with fewer artifacts.

Conclusion and Future Work

EulerMormer represents a significant step forward in Video Motion Magnification by being the first to leverage the global context of Transformers. By treating denoising as an integral part of the magnification process through dynamic filtering, it achieves a level of robustness and clarity previously unattainable with CNN-based approaches.

While highly effective, the computational complexity of Transformers remains a consideration for real-time applications. Future work may explore extending this framework to handle more complex "in-the-wild" videos where distinguishing between camera shake and subtle target motion remains a significant challenge.

Summary updated

Abstract

Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods. The source code is available at https://github.com/VUT-HFUT/EulerMormer.
BibTeX
@inproceedings{wang2024eulermormer,
  author = {Fei Wang and Dan Guo and Kun Li and Meng Wang},
  title = {{EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=BaXJRyuUtb},
}
Figure