When we think of motion, we naturally think of time. A video is a sequence of frames; a robotic trajectory is a series of motor commands; a human pose sequence is a list of joint coordinates tracking through seconds. However, modeling motion exclusively in the time domain introduces fundamental challenges: errors accumulate recursively, predictions often "freeze" into static poses, and high-frequency sensor noise can completely derail a controller.
To bypass these limitations, a powerful paradigm has emerged: representing motion not as a sequence of temporal coordinates, but as a combination of frequencies. By transforming trajectories into the frequency domain using tools like the Discrete Cosine Transform (DCT) or Fourier Transform, models gain a strong inductive bias that mirrors the physics of the real world.
Predicting the Whole Shape at Once
Why do time-domain models struggle with long-term forecasting? In a standard autoregressive model, predicting frame $t+1$ depends entirely on the accuracy of the prediction at frame $t$. A tiny error at the beginning cascades over time, leading to unnatural jitter or a complete collapse of motion.
Frequency-domain representations fundamentally change the objective. Instead of predicting coordinates frame-by-frame, a model can predict the coefficients of a spectral basis (such as a cosine or sine series). As shown by Cai et al. in Learning Progressive Joint Propagation for Human Motion Prediction, mapping joint trajectories to frequency coefficients allows a network to output the entire "shape" of a future motion sequence simultaneously. This global prediction eliminates recursive error accumulation entirely.
Furthermore, as noted by Barquero et al. in Didn't see that coming: a survey on non-verbal social human behavior forecasting, this spectral framing mathematically prevents the "freezing" artifacts common in temporal models, yielding smoother and more physiologically plausible trajectories. Bachmann et al. leverage this exact property in Motion Capture from Pan-Tilt Cameras with Unknown Orientation, optimizing a DCT basis to enforce physical smoothness and eliminate frame-by-frame jitter in 3D reconstructions.
Disentangling Global Actions from Fine Nuances
Physical motion is inherently hierarchical. When a person walks and waves, the walking motion is a slow, large-scale oscillation, while the hand wave consists of faster, localized movements. In the time domain, these signals are deeply entangled within the same coordinate stream.
In the frequency domain, however, they naturally separate. Low frequencies capture global, macro-level trajectories, while high frequencies capture fine-grained details and expressive nuances. Zhang et al. exploit this in WaMo: Wavelet-Enhanced Multi-Frequency Trajectory Analysis for Fine-Grained Text-Motion Retrieval by using wavelets to isolate broad actions (like walking) from subtle gestures (like a hand flick), significantly improving text-motion alignment.
This separation is equally powerful for facial animation and gesture generation. In JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation, Chakkera and Chatziagapi treat high-frequency localized mouth movements as speech-driven and low-frequency global movements as expression-driven. Similarly, Li et al. demonstrate in Audio2Gestures: Generating Diverse Gestures from Audio that frequency-domain modeling provides a mechanism for hierarchical control, allowing base postures and high-frequency expressive nuances to be edited independently.
Natural Noise Filtering and Data Efficiency
Physical systems—whether human muscles or robotic actuators—are bound by inertia. They cannot change state instantly, meaning true physical dynamics are inherently smooth and concentrated in low frequencies. High-frequency spikes in data are almost always noise, such as sensor jitter or shaky human demonstrations.
By shifting to the frequency domain, models can apply low-pass filters to isolate essential motion patterns. In Fourier Controller Networks for Real-Time Decision-Making in Embodied Learning, Tan et al. show that focusing on low-frequency components makes robot controllers highly resilient to high-frequency sensor noise, allowing them to achieve superior data efficiency compared to standard time-domain sequence models.
When learning from human demonstrations, Ming and Wang use the Discrete Fourier Transform in Stochastic Trajectory Optimization for Demonstration Imitation to strip away high-frequency jitters. To handle non-periodic paths without introducing boundary artifacts, the authors implement a back-stitching technique that makes the signal periodic, enabling smooth robotic execution that preserves the underlying shape of the human's intent.
Compressing High-Dimensional Action Spaces
Controlling complex systems, like a dense grid of actuators or a high-degree-of-freedom character, suffers from the curse of dimensionality. Learning an independent control strategy for every single coordinate over time is computationally exhausting.
Because physical motion is highly redundant, its representation in the frequency domain is remarkably sparse. A few low-frequency coefficients can accurately reconstruct a complex trajectory. Xue et al. leverage this sparsity in ArrayBot: Reinforcement Learning for Generalizable Distributed Manipulation through Touch by transforming a 256-dimensional actuator grid space into the frequency domain via a 2D DCT. This compresses the reinforcement learning search space, allowing the agent to coordinate collective patterns of movement rather than individual actuator states.
For generative tasks, Walker et al. apply a similar principle in An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders, using spectral trajectory compression to reduce high-dimensional pixel movements into a compact set of low-frequency coefficients, making future-frame prediction tractable.