TL;DR
Phase-PGF is a novel neural representation designed to capture "macro" (large, visually observable) motions in 2D and 3D scenes using low-dimensional, time-varying phases. By modulating a wavelet-based Gabor field with these phases, the method enables complex motion analysis and editing—such as magnification, smoothing, and loop detection—that was previously restricted to tiny "micro" motions.
The Challenge of Macro Motion
Motion analysis in computer vision has traditionally been divided into two distinct camps. One camp focuses on micro-motions (tiny vibrations or subtle changes), using phase-based filtering to magnify them. However, these methods break down when faced with large, "macro" motions like a person walking or a pendulum swinging.
The second camp uses dynamic neural representations (like D-NeRF) to model 3D scenes. While these can render high-quality videos, they represent motion through dense deformation or flow fields. These act as massive look-up tables tracking where every point moves, making it incredibly difficult to perform high-level analysis or edit the motion (e.g., smoothing out jitter or isolating a specific bouncing ball).
The authors bridge this gap by creating a representation that is both high-fidelity and mathematically interpretable for large-scale motion.
Phase-Based Neural Polynomial Gabor Fields
The core of the approach is the Phase-based neural polynomial Gabor field (Phase-PGF), which consists of several key components:
- Phase Generator: A neural network (specifically a Polynomial Neural Field) takes a timestamp $t$ and outputs a set of low-dimensional phases $h(t)$. These phases act as the "control knobs" for the scene's motion.
- Gabor Basis: The scene is represented spatially using Gabor functions—wavelets that are localized in both space and frequency.
- Modulation: The key innovation is that the time-varying phases $h(t)$ directly modulate the Gabor basis. This bakes the motion directly into the phase of the spatial representation.
- Neural Latent Decoder: To handle high-resolution details without exploding memory usage, Phase-PGF represents a low-resolution latent feature field. A separate 2D CNN decoder then upsamples these features into the final high-resolution image or 3D volume.
Key Insights and Capabilities
- Phases as Motion Proxies: The authors prove theoretically that if a motion is periodic, the learned phases will also be periodic. This allows the system to perform Fourier transforms on the phases rather than the pixels, making tasks like loop detection straightforward.
- Disentanglement via Frequency: Because the representation is wavelet-based, different motion components (e.g., a fast-moving ball vs. a slow-moving pendulum) naturally settle into different phase components. This enables unsupervised motion separation.
- Signal Processing for Editing: Since motion is represented as a 1D signal (the phase sequence), standard signal processing techniques apply. To smooth motion, one can apply a low-pass filter to the phases; to magnify it, simply multiply the phase by a constant.
Experimental Results
The authors evaluated Phase-PGF on a mix of 2D and 3D data, including synthetic videos with ground-truth trajectories and real-world internet videos.
- Interpretability: In human preference studies, Phase-PGF's learned phases were judged to better explain scene motion 79.2% of the time compared to previous state-of-the-art methods like Phase-NIVR.
- Motion Magnification: Phase-PGF successfully magnified macro motions with significantly fewer artifacts (ghosting or flickering) than traditional Eulerian or neural baselines. It achieved a much lower Fréchet Inception Distance (FID) of 0.512 compared to Phase-NIVR's 10.255 on manipulated videos.
- 3D Motion Separation: The model successfully isolated and edited the motion of one object in a 3D scene without affecting others, simply by filtering the phase components associated with that object's frequency.
Limitations and Future Work
While a significant step forward, Phase-PGF has limitations. "Boldly" magnifying very large motions can still introduce slight artifacts. Additionally, the current implementation uses a fixed number of Gabor bases, making it difficult to scale to highly complex 3D environments. The authors suggest exploring "spatially adaptive" Gabor bases in the future. Finally, the model struggles with chaotic or shapeless motions (like blowing hair or fire), where it can only capture a coarse tendency rather than precise dynamics.
Summary updated