Disentangling Appearance and Motion in Video Representations

139 concepts121 papers1 proposal

When humans watch a video of a person skiing, we effortlessly separate the skier's physical actions from the static, snow-covered mountain. We can easily imagine the exact same skiing motion performed on a grassy hill, or a completely different action—like walking—performed on that same snowy mountain.

For deep neural networks, however, this separation is far from automatic. Standard spatiotemporal models treat video as a monolithic block of pixels, inextricably entangling appearance (the static textures, backgrounds, and object identities) with motion (the dynamic changes and trajectories over time). This spatiotemporal entanglement introduces severe bottlenecks across both video understanding and video generation.

The Two Sides of Entanglement: Cheating and Leakage

To understand why researchers are determined to decouple appearance from motion, we must look at how entanglement degrades performance in two core areas: video recognition and video generation.

1. The "Shortcut" Problem in Video Recognition

In video classification, networks are notorious for taking the path of least resistance. If a model can identify an action like "swimming" purely by detecting the static texture of blue pool water, it will completely ignore the actual temporal dynamics of the swimmer.

Kensho Hara et al. demonstrate this bias in Rethinking Training Data for Mitigating Representation Biases in Action Recognition, showing that standard 3D convolutional neural networks (CNNs) heavily rely on static scene context rather than true motion. When a model cannot separate the "where" of the environment from the "what" of the action, it fails to generalize to new environments.

To expose this flaw, the authors of Is Appearance Free Action Recognition Possible? constructed the Appearance Free Dataset (AFD), where individual frames consist of random noise, yet clear actions emerge when viewed as a sequence. While humans excel at recognizing these pure motion patterns, standard video models fail significantly because they lack explicit mechanisms to isolate optical flow from static shapes and colors.

2. The "Appearance Leakage" Problem in Video Generation

The consequences of entanglement are equally severe when generating or editing videos. When a text-to-video diffusion model is tasked with learning a specific motion pattern from a reference video—such as a unique dance step—and applying it to a new character, it often suffers from appearance leakage. Instead of transferring just the movement, the model accidentally copies the textures, clothing, and background of the source video.

As noted by Hyeonho Jeong et al. in VMC: Video Motion Customization Using Temporal Attention Adaption for Text-to-Video Diffusion Models, this occurs because generative models lack a dedicated, isolated representation for motion that exists independently of static frame content. Without this separation, modifying an object's visual attributes inadvertently corrupts its motion trajectory.


Architectural Disentanglement: Separating Space and Time

The first major strategy to combat this problem is rewriting the network architecture itself to process space and time through distinct, parallel pipelines.

Instead of using standard 3D convolutions that conflate appearance and motion within their internal representations, Limin Wang 0002 et al. introduced the SMART block in Appearance-and-Relation Networks for Video Classification. The SMART block explicitly decomposes video processing into an appearance stream and a relation stream, using multiplicative interactions to capture temporal dynamics directly from raw RGB frames without relying on expensive external optical flow computations.

Raw Video Frame ───► [ Spatial Pipeline (Appearance/Textures) ] ───┐
                                                                   ├──► Disentangled Feature
Raw Video Frame ───► [ Temporal Pipeline (Motion/Relations) ] ────┘

From a optimization perspective, Xianhang Li argues in In Defense of Image Pre-Training for Spatiotemporal Recognition that standard 3D convolutions prevent models from independently optimizing for what an object looks like versus how it moves. By introducing Spatial-Temporal Separable (STS) convolutions, the architecture forces a structural split between spatial feature extraction and temporal modeling, allowing models to leverage powerful static image pre-training without entangling the subsequent motion signals.


Latent Factorization: Isolating Motion in Diffusion Models

In modern generative architectures like Diffusion Transformers (DiTs) and text-to-video models, researchers achieve disentanglement by factorizing the model's latent space or internal attention layers.

For instance, Yang Zhou et al. developed a one-shot motion customization framework in Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models that utilizes a dedicated Appearance Absorber to isolate style and texture, alongside a Temporal LoRA to capture pure motion dynamics. This prevents the model from memorizing the specific backgrounds of a reference video, ensuring high-fidelity motion transfer to entirely new subjects.

Training-Free Motion Extraction

Remarkably, recent work shows that this disentanglement can be achieved without retraining models. Zeqi Xiao et al. discovered in Video Diffusion Models are Training-free Motion Interpreter and Controller that video diffusion models inherently encode motion within their internal features, but the signal is buried under dominant content data.

By applying a simple training-free subtraction and filtering process across consecutive denoising steps, they isolate what they term MOtion FeaTures (MOFT). Because MOFT strips away the underlying object identities, it can be used as a precise control signal to steer the motion of completely different characters across various architectures without causing appearance leakage.


Self-Supervised Decoupling: Forcing Models to Focus on Dynamics

When labeled data is unavailable, how can a network be forced to learn separate representations for context and motion? The solution lies in designing self-supervised contrastive learning objectives that break the natural correlations between scenes and actions.

Long-Short View Decomposition

Mohsen Fayyaz et al. introduce a feature decomposition method in Long Short View Feature Decomposition via Contrastive Video Representation Learning based on temporal scale. They contrast "long views" of a video (which capture stationary background context over extended periods) with "short views" (which capture rapid, non-stationary temporal changes). By forcing the network to distinguish between what remains stable over long horizons and what changes in the short term, the model naturally learns to segregate static context from dynamic action.

Long Video View  ───► [ Extracts Stationary Features ] ───► Background Context
Short Video View ───► [ Extracts Non-Stationary Features ] ─► Pure Motion Dynamics

Breaking Background Shortcuts via Warping

To explicitly prevent models from "cheating" by looking at the environment, Alex Jinpeng Wang et al. proposed the DSM (Decoupling the Scene and the Motion) framework in Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion.

DSM applies artificial spatial warping and temporal disturbances to video clips to create training pairs where the background scene is distorted but the underlying motion dynamics remain identifiable. This breaks the reliable correlation between an action and its environment, forcing the network's internal representations to focus on the pure motion signal to solve the self-supervised task.

By decoupling these variables, modern video models move away from superficial texture-matching and toward a deeper, more robust understanding of spatiotemporal physics.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_content_motion_disentanglement_537b00bc,
  title = {content-motion disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/content-motion-disentanglement/rdr_537b00bcf9614b9fbd87e1db7123e105},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_content_motion_disentanglement_537b00bc_v1,
  title = {content-motion disentanglement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/content-motion-disentanglement/rdr_537b00bcf9614b9fbd87e1db7123e105/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake