Decoupling Motion and Appearance in Video Diffusion

This direction is split into other directions: temporal prior preservation. Its last version stays readable and citable here.

Version 1 of 144 papers58 conceptsfrozen 2026-10-04

Generating a video is fundamentally more difficult than generating an image because it requires two distinct types of consistency: spatial and temporal. A model must not only understand what a subject looks like (appearance) but also how that subject evolves over time (motion).

Recent research has converged on a powerful insight: in modern video diffusion models, these two properties are functionally decoupled within the architecture. By selectively manipulating spatial or temporal layers, researchers can customize a video's motion without changing its subjects, or vice versa, effectively solving the problem of "appearance leakage."

The Factorization of Video Architecture

Most state-of-the-art video models are built by "inflating" a pre-trained image model. As Jinbo Xing and colleagues describe in Make-Your-Video, this involves taking a frozen spatial backbone—which already knows how to draw textures and objects—and inserting new temporal modules.

This architectural split creates a natural division of labor. The spatial layers (often 2D convolutions and cross-attention) handle the "what," while the temporal layers (1D temporal attention or recurrent blocks) handle the "when." Rohit Girdhar and the team behind Emu Video demonstrate that by factorizing the process—generating a high-quality image first and then animating it—one can leverage the massive prior knowledge of image models without degrading their quality during video training.

Preserving Identity by Freezing Spatial Layers

When we want to teach a model a new motion—like a specific dance or a camera "dolly zoom"—the risk is that the model will also learn the appearance of the person in the reference video. This is known as appearance leakage.

To combat this, Hyeonho Jeong and Geon Yeong Park introduced VMC, a framework that fine-tunes only the temporal attention layers. By keeping the spatial and cross-attention layers fixed, the model is physically unable to "save" new visual information about the subject's identity. It can only update its understanding of how pixels shift between frames.

Similarly, Yang Zhou and Jimei Yang use Customize-A-Video to apply Low-Rank Adaptation (LoRA) exclusively to temporal cross-frame attention. This focuses the learning capacity on inter-frame relationships, allowing a user to transfer a monkey's movement to a fictional character without the character turning into a monkey.

Preserving Motion by Freezing Temporal Layers

The inverse problem occurs when we want to introduce a new subject (like a specific pet) into a video model. If we fine-tune the whole model on images of the pet, it might "forget" how things move, leading to flickering or static videos.

Zhao Wang and Qi Dou show in CustomVideo that freezing the temporal layers while updating only the spatial cross-attention weights allows the model to learn new subjects while perfectly retaining its pre-trained motion priors. This ensures that the new subject moves with the same fluid realism as the original training data. Yutao Cui applies a similar logic in MoG, arguing that frozen temporal layers act as a "motion anchor" that prevents the model from generating physically implausible sequences when adapting to new visual domains.

Why Temporal Attention is the Key

Why is the temporal attention layer the primary site for motion? In a Transformer-based video model, temporal attention allows a pixel at coordinates $(x, y)$ in frame $t$ to "look" at the same coordinates in frames $t-1$ and $t+1$.

Pan Zhang and Pengyang Ling discovered in MotionClone that these temporal attention weights actually serve as a sparse representation of the video's motion. By "cloning" these weights, you can transfer movement without any training at all. This is why Hao He and Bo Dai chose to inject camera control signals specifically into these layers in CameraCtrl; it influences the "flow" of the video without touching the "paint" of the individual frames.

The Temporal Dimension of Denoising

The decoupling of motion and appearance isn't just architectural—it's also temporal. Yixuan Ren and Abhinav Shrivastava found that motion is predominantly encoded during the early, high-noise stages of the diffusion denoising process.

In Characterizing Motion Encoding in Video Diffusion Timesteps, they show that by restricting fine-tuning to these specific timesteps, one can capture complex dynamics while the model remains "blind" to the fine-grained appearance details that emerge later in the denoising process. This provides a second layer of protection against identity drift, ensuring that the model learns the path of the movement rather than the pixels of the mover.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupling_motion_and_appearance_in_vide_d1d111a3,
  title = {Decoupling Motion and Appearance in Video Diffusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-motion-and-appearance-in-video-diffusion/rdr_d1d111a335e84ff3a9e48254ea60f2d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupling_motion_and_appearance_in_vide_d1d111a3_v1,
  title = {Decoupling Motion and Appearance in Video Diffusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-motion-and-appearance-in-video-diffusion/rdr_d1d111a335e84ff3a9e48254ea60f2d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}