Spatio-Temporal Decoupling for Video Model Adaptation

22 concepts16 papers1 proposal

When adapting deep learning models to video tasks, researchers face a fundamental challenge: videos are not just collections of static images, nor are they abstract motion vectors. They are a complex entanglement of spatial information (what things look like, their style, geometry, and identity) and temporal information (how things move, accelerate, and interact over time).

If you fine-tune a model on a video dataset using a uniform learning rate across all layers, a destructive tug-of-war occurs. As Liu et al. observed in the development of the Video Swin Transformer, standard optimization can cause a model to discard its highly optimized spatial representations before it ever successfully learns to process motion.

To solve this, a powerful consensus has emerged across video generation, editing, and tracking: spatio-temporal parameter decoupling. By freezing one dimension (space or time) while selectively updating or adding modules in the other, models can acquire new capabilities without destroying their hard-earned priors.


Freezing Space to Learn Time: Image-to-Video Adaptation

The most common application of this concept occurs when transforming a powerful static image foundation model into a video model. Image models trained on billions of frames possess an extraordinary understanding of visual aesthetics, textures, and open-domain concepts. If we update these spatial weights while trying to teach the model how time works, we trigger "concept forgetting"—the degradation of the model's pristine image generation capabilities.

Standard Fine-tuning:
[Spatial Weights] ──(Updates)──> Loss of Visual Aesthetics
[Temporal Weights] ──(Updates)──> Learning Motion

Decoupled Fine-tuning:
[Spatial Weights] ──(FREEZING)──> Preserves Visual Aesthetics
[Temporal Weights] ──(Updates)──> Learning Motion

To prevent this, frameworks like Emu Video (Girdhar et al.) and Make-Your-Video (Liu et al.) explicitly freeze the spatial layers of pre-trained image generators. They insert newly initialized temporal layers or auxiliary modules between the spatial blocks. Because the spatial layers are frozen, the model's capacity is entirely focused on learning inter-frame motion dynamics, ensuring that the final video retains the exact stylistic diversity and high-fidelity details of the original image model.

This principle extends beyond generation to dense spatial tasks. In Video Depth Anything, Guo et al. freeze a pre-trained image encoder entirely while training a dedicated spatio-temporal head. This ensures that the model's deep understanding of single-frame scene geometry remains uncorrupted while it learns to smooth depth predictions consistently across long video sequences.


Freezing Time to Learn Space: Domain and Subject Customization

The inverse problem arises when you already have a high-quality video foundation model that understands physical motion, but you want to adapt it to a new visual domain—such as a specific character's geometry or an artistic style like cartoons.

Real-world video models implicitly learn complex motion physics (gravity, fluid dynamics, dis-occlusions). If you fine-tune the entire model on a small dataset of cartoon clips, the model will quickly forget how real-world objects move.

Live-Action Video Model ───> [Freeze Temporal Layers] ───> Fine-tune Spatial Layers ───> Cartoon Video Model
                               (Preserves Physics)             (Learns Style/Texture)

To transfer real-world motion priors to stylized domains, ToonCrafter (Xing et al.) freezes the temporal layers of a live-action video diffusion model while fine-tuning only the spatial layers and context projectors. Similarly, ToonComposer (Li et al.) introduces dimension-constrained adapters that restrict optimization updates strictly to spatial components. This constraint allows the model to render cartoon textures and sketches while maintaining the sophisticated, physically plausible motion dynamics inherent in the base video model.

A parallel strategy is used for character and subject customization. In AnyMoLe (Yun et al.), the authors adapt video models to specific synthetic characters by freezing the temporal modules and fine-tuning spatial modules on a short context clip. This bridges the domain gap in appearance without losing the model's generalized understanding of motion physics.

For multi-subject video generation, CustomVideo (Wang et al.) freezes temporal layers and updates only the spatial cross-attention weights. This allows the model to bind new visual identities (like a specific pet) to the scene without requiring expensive multi-subject video datasets or sacrificing temporal consistency.


Isolating Motion to Prevent Appearance Leakage

A third facet of this architecture occurs during motion customization or video editing, where the goal is to inject a specific camera movement or action trajectory without altering the visual identity of the subjects. If spatial layers are allowed to change during this process, a phenomenon known as appearance leakage occurs: the model accidentally bakes the identity, colors, or textures of the reference motion source into the target video.

To isolate motion learning from appearance, VMC (Jeong et al.) freezes all spatial and cross-attention layers, restricting fine-tuning exclusively to temporal attention layers. By forcing the network to minimize a frame-to-frame residual loss solely through temporal weights, the model captures the dynamics of a reference video while leaving its generative diversity untouched.

Reference Video (Motion Source) ───> [Fine-tune ONLY Temporal Attention] ───> Target Video
                                          │
                                    [Freeze Spatial] ───> Prevents Appearance Leakage

Similarly, Customize-A-Video (Zhou et al.) applies Low-Rank Adaptation (LoRA) exclusively to cross-frame temporal attention layers, focusing the learning capacity entirely on inter-frame relationships.

This exact mechanism makes precise camera control possible. In CameraCtrl, He et al. inject camera trajectory embeddings exclusively into the temporal attention layers of video diffusion models. Because the spatial layers are bypassed, the control signal modifies how the frames transition relative to one another (simulating a camera pan or zoom) without inadvertently altering the stylistic or structural content of individual frames.

Even in discriminative or tracking tasks, this decoupling acts as a safeguard. For instance, SeqHAND (Chang et al.) detaches and freezes recurrent temporal layers when adapting a 3D hand pose estimator to new static image domains, ensuring that the underlying motion tracking logic remains uncorrupted by domain shifts in raw pixel appearance.

Summary of Decoupling Strategies

Objective What to Freeze What to Tune / Add Key Benefit
Image-to-Video Spatial Layers Temporal Layers / Heads Preserves visual aesthetics; prevents concept forgetting.
Domain Adaptation Temporal Layers Spatial Layers / Adapters Retains physical motion priors while changing style/geometry.
Motion Customization Spatial Layers Temporal Attention / LoRA Prevents appearance leakage; isolates motion from identity.

Key Foundational Work

SeqHAND: RGB-Sequence-Based 3D Hand Pose and Shape Estimation (2020) — contributed 1 concept elements to this direction.
Video Swin Transformer (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_prior_preservation_adbffe91,
  title = {temporal prior preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-prior-preservation/rdr_adbffe91cf93439c961aa3a3e9f12fac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_prior_preservation_adbffe91_v1,
  title = {temporal prior preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-prior-preservation/rdr_adbffe91cf93439c961aa3a3e9f12fac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake