Abstract
Implicit neural representation (INR) has been successful in representing static images. Contemporary image-based INR, with the use of Fourier-based positional encoding, can be viewed as a mapping from sinusoidal patterns with different frequencies to image content. Inspired by that view, we hypothesize that it is possible to generate temporally varying content with a single image-based INR model by displacing its input sinusoidal patterns over time. By exploiting the relation between the phase information in sinusoidal functions and their displacements, we incorporate into the conventional image-based INR model a phase-varying positional encoding module, and couple it with a phase-shift generation module that determines the phase-shift values at each frame. The model is trained end-to-end on a video to jointly determine the phase-shift values at each time with the mapping from the phase-shifted sinusoidal functions to the corresponding frame, enabling an implicit video representation. Experiments on a wide range of videos suggest that such a model is capable of learning to interpret phase-varying positional embeddings into the corresponding time-varying content. More importantly, we found that the learned phase-shift vectors tend to capture meaningful temporal and motion information from the video. In particular, manipulating the phase-shift vectors induces meaningful changes in the temporal dynamics of the resulting video, enabling non-trivial temporal and motion editing effects such as temporal interpolation, motion magnification, motion smoothing, and video loop detection.
Introduction
Implicit Neural Representations (INRs) have revolutionized visual data synthesis by modeling images as continuous functions, typically mapping spatial coordinates $(x, y)$ to color values. Extending this to video has traditionally involved treating time as a third coordinate $(x, y, t)$, effectively viewing video as a static 3D volume. However, this "naive" approach lacks explicit modeling of temporal dynamics, making it difficult to manipulate motion post-training.
This paper introduces a "motion-adjustable" INR that treats motion as a steerable parameter. By encoding temporal dynamics as phase shifts within Fourier-based positional encodings, the model allows for complex video editing tasks—such as motion magnification and smoothing—without retraining.
Methodology
The core innovation is a phase-varying positional encoding module. Standard INRs use Fourier-based Positional Encoding (PE) to map coordinates to high-dimensional sinusoidal features. The proposed architecture consists of two coupled modules:
- Phase-Shift Generation Module ($M_p$): A network that maps time $t$ to a set of phase-shift vectors $\phi(t)$.
- Frame Generation Module ($M_f$): An image-based INR that incorporates these phase shifts into its sinusoidal inputs: $\sin(2^i \pi c + \phi_i(t))$.
Mathematically, adding a phase shift to a sine wave is equivalent to a displacement. By learning these shifts, the model learns to displace underlying sinusoidal patterns to reconstruct moving content. To ensure the model learns a structured phase space rather than simple memorization, an asymmetric training strategy is employed: the full model is updated on half the frames, while for the remainder, the frame generator is frozen, forcing the phase generator to find shifts compatible with existing spatial features.
Results and Applications
The model was evaluated on the WAIC-TSR dataset and custom real-world sequences. It achieves reconstruction quality comparable to standard 3D coordinate-based INRs (PSNR of 32.05 vs. 31.98) while offering unique editing capabilities.
Key applications enabled by this representation include:
- Motion Magnification: Amplifying subtle vibrations by scaling phase-shift differences.
- Motion Smoothing: Stabilizing jitter by low-pass filtering the phase-shift sequence.
- Temporal Interpolation: Generating intermediate frames by interpolating in the phase-shift space.
- Loop Detection: Identifying looping points by comparing phase-shift vectors.
Conclusion and Future Work
While the model provides a steerable representation of video motion, it currently requires significant training time (approx. 18 hours per video) and can exhibit minor spectral bias artifacts. Future research may explore meta-learning or hypernetworks to generalize the model across multiple videos, potentially reducing the per-video optimization overhead.
Summary updated