When humans watch a video or observe a moving object, we naturally separate what the object is from how it is moving. A running cat retains its "cat-ness"—its geometry, fur texture, and unique identity—even as its spatial coordinates shift rapidly through time.
However, standard deep learning architectures operating on spatiotemporal data (such as video frames or 3D point clouds) often treat space and time as an entangled, monolithic block. This entanglement forces networks to learn high-dimensional representations where every change in pose or motion looks like an entirely new object. The result is severe computational inefficiency, blurry predictions, and a lack of granular control.
To overcome this, a prominent research direction focuses on spatiotemporal factorization: explicitly decoupling static spatial attributes (geometry, appearance, identity) from dynamic temporal attributes (motion, pose, deformation). By splitting these features into orthogonal latent spaces, models can process, generate, and track dynamic data with unprecedented efficiency and fidelity.
The Cost of Entanglement
Why is treating space and time as a single entity problematic? In generative tasks, forcing a network to simultaneously synthesize fine visual details and fluid motion creates a fundamental optimization conflict. As Ohnishi et al. observed in Hierarchical Video Generation From Orthogonal Information: Optical Flow and Texture, models trained on coupled representations often trade image clarity for realistic movement, resulting in blurry or distorted frames.
Entangled Approach:
[Input Video] ---> [Single Spatiotemporal Latent Space] ---> [Blurry/Unstable Output]
(Optimization Trade-off)
Factorized Approach:
---> [Static Spatial Latent (Identity/Texture)]
[Input Video] ---> [Sharp, Controlled Output]
---> [Dynamic Temporal Latent (Pose/Motion)]
This issue is amplified in 3D and 4D geometric domains, such as LiDAR point cloud processing. Standard 4D convolutions suffer from massive data redundancy because they re-process local geometric structures across overlapping temporal windows (Wang et al. in PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition). Furthermore, when geometry and motion are entangled, optimization algorithms can become biased toward one feature type over the other. For instance, high-frequency motion noise can easily corrupt a model's understanding of an object's underlying physical shape (Lin et al. in SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking).
Mechanisms of Spatiotemporal Factorization
To mathematically separate these features, modern networks rely on two primary architectural strategies: dual-encoder latent partitioning and canonical geometric anchoring.
Dual-Encoder Latent Partitioning
In video processing, models split their bottleneck representations into distinct pathways. One branch extracts time-invariant features (like identity or style), while the other captures time-varying features (like optical flow or keypoint velocities).
For example, DRNET (Denton et al. in Unsupervised Learning of Disentangled Representations from Video) and RED-Net (Peng et al. in RED-Net: A Recurrent Encoder-Decoder Network for Video-based Face Alignment) utilize dual encoders to isolate content from pose. To ensure that the motion branch does not accidentally encode identity information, researchers often apply adversarial losses or permutation-invariant constraints (Franceschi et al. in Stochastic Latent Residual Video Prediction; Yang et al. in Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis). This forces the temporal encoder to focus exclusively on residual changes between frames.
Canonical and Local Geometric Anchoring
In 3D and 4D domains, factorization is achieved by anchoring motion to a local reference frame. The LeaF framework (Liu et al. in LeaF: Learning Frames for 4D Point Cloud Sequence Understanding) factorizes point clouds by attaching local coordinate frames to moving objects. This isolates motion-invariant geometric features from global trajectories, drastically simplifying the learning process.
Similarly, Niemeyer et al. in Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics use separate encoders to model static 3D geometry and continuous spatio-temporal vector fields independently, allowing the network to learn shape characteristics and movement trajectories as isolated mathematical functions.
Why Factorization Works: Modular Control and Invariance
Decoupling space and time provides two massive advantages that entangled models cannot match: modular control and environmental invariance.
Modular Control and Motion Transfer
Once appearance and motion are mapped to independent vectors, they can be mixed and matched arbitrarily. A static image of a person can be animated using the motion trajectory or skeletal pose sequence of an entirely different video (Yang et al. in Pose Guided Human Video Generation; Kim et al. in Video Prediction with Appearance and Motion Conditions).
In 4D generation, this allows learned motion sequences to be retargeted to novel static geometries that were never seen during training (Han et al. in Learning Neural Deformation Representation for 4D Dynamic Shape Generation; Zhang et al. in DNF: Unconditional 4D Generation with Dictionary-based Neural Fields). Generative models can treat visual features as a "texture library," rotating or translating objects without altering their underlying identity (Wu et al. in Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models).
Invariance and Robustness
Factorization makes models highly resilient to domain shifts and environmental noise. In medical imaging, restricting temporal tracking exclusively to structural latent factors ensures that a model's understanding of motion (e.g., a beating heart) remains invariant to changes in imaging modalities or style (Valvano et al. in Temporal Consistency Objectives Regularize the Learning of Disentangled Representations).
This principle even extends to biological data analysis. Song et al. demonstrated in Classification of Spatiotemporal Neural Activity Patterns in Brain Imaging Data that separating the physical footprint of neural activity from its movement trajectory allows for the classification of distinct biological states that share identical spatial shapes but exhibit entirely different temporal behaviors. By breaking the spatiotemporal problem into its constituent parts, networks avoid the curse of dimensionality and achieve far greater generalization.
Key Foundational Work
Video Prediction with Appearance and Motion Conditions (2018) — contributed 2 concept elements to this direction.
Hierarchical Video Generation From Orthogonal Information: Optical Flow and Texture (2018) — contributed 2 concept elements to this direction.
Classification of Spatiotemporal Neural Activity Patterns in Brain Imaging Data (2018) — contributed 2 concept elements to this direction.