Decoupling Appearance and Motion in Generative Video Models

This direction is split into other directions: decoupled spatial-temporal training. Its last version stays readable and citable here.

Version 1 of 1180 papers216 conceptsfrozen 2026-10-04

Generating high-fidelity video is a dual challenge: a model must understand the intricate textures of the physical world (spatial) and the fluid laws of how those textures change over time (temporal). Attempting to learn both simultaneously often leads to a "jack of all trades, master of none" scenario, where models produce blurry textures or physically impossible movements.

To solve this, researchers have converged on a shared strategy: staged training curricula that decouple spatial appearance from temporal dynamics. By mastering "what" an object is before learning "how" it moves, these models achieve higher stability, better identity preservation, and greater data efficiency.

The Spatial-First Intuition

The core intuition behind this research direction is that spatial representation is the foundation upon which motion is built. If a model cannot reliably generate a static human face or a car, it cannot hope to animate them consistently across frames.

In VideoCrafter2, Haoxin Chen and colleagues demonstrate that motion and appearance can be learned from entirely different data sources. They use low-quality videos to learn motion priors and high-quality images to refine visual aesthetics. This separation allows the model to inherit the rich textures of professional photography without being limited by the lower resolution of available video datasets.

Similarly, TiVGAN by Doyeon Kim employs an evolutionary approach, mastering single-frame generation before gradually increasing the temporal complexity. This "step-by-step" mastery prevents the optimization process from being overwhelmed by the high-dimensional complexity of raw video.

Freezing the Canvas to Animate the Brush

Once a model has learned a robust spatial representation, the next phase involves introducing motion. A recurring mechanism across these papers is the use of frozen spatial weights. By locking the layers responsible for identity and texture, researchers can train dedicated temporal modules without risking "catastrophic forgetting" of visual details.

For instance, in ManiVideo, Ruizhi Shao and others generate complex hand-object manipulations by first optimizing spatial weights on static images and then freezing them to train temporal layers. This ensures that the "ghost finger" artifacts common in video generation are minimized, as the model's understanding of hand geometry remains fixed while it learns the timing of a grasp.

This decoupling also solves the "attribute binding" problem. In VideoDreamer, Hong Chen shows that separating identity modules from motion modules prevents the visual features of a motion source (like a reference video) from "leaking" into the subject being animated. This allows a user to apply a specific dance to a specific character without the character's face morphing into the dancer's.

Geometric Scaffolding for 4D Consistency

In the realm of 3D and 4D (dynamic 3D) generation, this decoupling takes the form of canonical representations. Instead of learning a video as a sequence of independent pixels, models like Control4D and Stag-1 use a static 3D "scaffold."

Ruizhi Shao uses a structured representation called GaussianPlanes to establish a static canonical version of a portrait. By editing this static version first and then optimizing the deformation field (the movement) in a second stage, the system achieves spatiotemporal consistency up to 60x faster than end-to-end methods. The static scaffold acts as a "ground truth" for appearance, ensuring that as the person moves, their features don't drift or disappear.

Bridging the Data Gap

A significant advantage of decoupled training is its ability to leverage "in-the-wild" data that lacks full annotation. Make-An-Animation by Samaneh Azadi overcomes the scarcity of high-quality 3D motion data by pre-training on 35 million "pseudo-poses" extracted from static images. By first learning the vocabulary of human poses from images, the model only needs a smaller, specialized dataset to learn how those poses transition over time.

This same logic applies to camera control. CETCAM and HumanVid use two-phase training to separate the acquisition of motion dynamics from visual refinement. Zelin Zhao argues that this allows models to learn general camera movement from massive amounts of unannotated video before being "polished" on high-quality, curated datasets for final output.

Summary of the Staged Approach

The shift toward decoupled training reflects a maturing understanding of the video generation task. Rather than treating video as a monolithic block of data, the field is moving toward a modular curriculum:

  1. Spatial Phase: Establish high-fidelity identity, texture, and geometry using images or canonical 3D representations.
  2. Temporal Phase: Introduce motion, camera trajectories, and transitions by training dedicated layers while keeping spatial knowledge stable.
  3. Refinement Phase: Optionally fine-tune both together to ensure the "seams" between appearance and motion are invisible.

By breaking the problem down, researchers like Yixuan Li and Zhenzhou Fang are producing models that are not only more realistic but also more controllable, allowing for precise "cinematography" in AI-generated content.

Papers in this version

Showing 60 of 180 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupling_appearance_and_motion_in_gene_17827283,
  title = {Decoupling Appearance and Motion in Generative Video Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-appearance-and-motion-in-generative-video-models/rdr_178272830f974d48b1cf07aeb940150f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupling_appearance_and_motion_in_gene_17827283_v1,
  title = {Decoupling Appearance and Motion in Generative Video Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-appearance-and-motion-in-generative-video-models/rdr_178272830f974d48b1cf07aeb940150f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}