Neural Polynomial Gabor Fields for Macro Motion Analysis

Chen Geng, Hong-Xing Yu, Sida Peng, Xiaowei Zhou, Jiajun Wu

TL;DR

Phase-PGF is a novel neural representation designed to capture "macro" (large, visually observable) motions in 2D and 3D scenes using low-dimensional, time-varying phases. By modulating a wavelet-based Gabor field with these phases, the method enables complex motion analysis and editing—such as magnification, smoothing, and loop detection—that was previously restricted to tiny "micro" motions.

The Challenge of Macro Motion

Motion analysis in computer vision has traditionally been divided into two distinct camps. One camp focuses on micro-motions (tiny vibrations or subtle changes), using phase-based filtering to magnify them. However, these methods break down when faced with large, "macro" motions like a person walking or a pendulum swinging.

The second camp uses dynamic neural representations (like D-NeRF) to model 3D scenes. While these can render high-quality videos, they represent motion through dense deformation or flow fields. These act as massive look-up tables tracking where every point moves, making it incredibly difficult to perform high-level analysis or edit the motion (e.g., smoothing out jitter or isolating a specific bouncing ball).

The authors bridge this gap by creating a representation that is both high-fidelity and mathematically interpretable for large-scale motion.

Phase-Based Neural Polynomial Gabor Fields

The core of the approach is the Phase-based neural polynomial Gabor field (Phase-PGF), which consists of several key components:

  1. Phase Generator: A neural network (specifically a Polynomial Neural Field) takes a timestamp $t$ and outputs a set of low-dimensional phases $h(t)$. These phases act as the "control knobs" for the scene's motion.
  2. Gabor Basis: The scene is represented spatially using Gabor functions—wavelets that are localized in both space and frequency.
  3. Modulation: The key innovation is that the time-varying phases $h(t)$ directly modulate the Gabor basis. This bakes the motion directly into the phase of the spatial representation.
  4. Neural Latent Decoder: To handle high-resolution details without exploding memory usage, Phase-PGF represents a low-resolution latent feature field. A separate 2D CNN decoder then upsamples these features into the final high-resolution image or 3D volume.
The Phase-PGF architecture: a phase generator modulates a Gabor basis to create a dynamic feature field, which is then decoded into a final rendering.

Key Insights and Capabilities

  • Phases as Motion Proxies: The authors prove theoretically that if a motion is periodic, the learned phases will also be periodic. This allows the system to perform Fourier transforms on the phases rather than the pixels, making tasks like loop detection straightforward.
  • Disentanglement via Frequency: Because the representation is wavelet-based, different motion components (e.g., a fast-moving ball vs. a slow-moving pendulum) naturally settle into different phase components. This enables unsupervised motion separation.
  • Signal Processing for Editing: Since motion is represented as a 1D signal (the phase sequence), standard signal processing techniques apply. To smooth motion, one can apply a low-pass filter to the phases; to magnify it, simply multiply the phase by a constant.
By analyzing the learned phases, the system maps 1D periodic signals directly to the macro motion of objects, enabling tasks like seamless loop detection.

Experimental Results

The authors evaluated Phase-PGF on a mix of 2D and 3D data, including synthetic videos with ground-truth trajectories and real-world internet videos.

  • Interpretability: In human preference studies, Phase-PGF's learned phases were judged to better explain scene motion 79.2% of the time compared to previous state-of-the-art methods like Phase-NIVR.
  • Motion Magnification: Phase-PGF successfully magnified macro motions with significantly fewer artifacts (ghosting or flickering) than traditional Eulerian or neural baselines. It achieved a much lower Fréchet Inception Distance (FID) of 0.512 compared to Phase-NIVR's 10.255 on manipulated videos.
  • 3D Motion Separation: The model successfully isolated and edited the motion of one object in a 3D scene without affecting others, simply by filtering the phase components associated with that object's frequency.

Limitations and Future Work

While a significant step forward, Phase-PGF has limitations. "Boldly" magnifying very large motions can still introduce slight artifacts. Additionally, the current implementation uses a fixed number of Gabor bases, making it difficult to scale to highly complex 3D environments. The authors suggest exploring "spatially adaptive" Gabor bases in the future. Finally, the model struggles with chaotic or shapeless motions (like blowing hair or fire), where it can only capture a coarse tendency rather than precise dynamics.

Summary updated

Abstract

We study macro motion analysis, where macro motion refers to the collection of all visually observable motions in a dynamic scene. Traditional filtering-based methods on motion analysis typically focus only on local and tiny motions, yet fail to represent large motions or 3D scenes. Recent dynamic neural representations can faithfully represent motions using correspondences, but they cannot be directly used for motion analysis. In this work, we propose Phase-based neural polynomial Gabor fields (Phase-PGF), which learns to represent scene dynamics with low-dimensional time-varying phases. We theoretically show that Phase-PGF has several properties suitable for macro motion analysis. In our experiments, we collect diverse 2D and 3D dynamic scenes and show that Phase-PGF enables dynamic scene analysis and editing tasks including motion loop detection, motion factorization, motion smoothing, and motion magnification. Project page: https://chen-geng.com/phasepgf
BibTeX
@inproceedings{geng2024neural,
  author = {Chen Geng and Hong-Xing Yu and Sida Peng and Xiaowei Zhou and Jiajun Wu},
  title = {{Neural Polynomial Gabor Fields for Macro Motion Analysis}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=dTlKCQuuxP},
}
Figure