Bridging the Pose to Video Realism Gap

23 concepts23 papers1 proposal

When generating human-centric video from abstract inputs like text or speech, generative models face an immense search space. Trying to predict raw pixels directly across time often results in a chaotic "pixel soup" where structural consistency is lost. To make this problem tractable, researchers widely adopt a two-stage decoupled paradigm: first, translate the high-level semantic input into a low-dimensional structural guide (such as skeletal joint coordinates), and second, render that structural guide into a photorealistic video.

While this separation simplifies the learning process, it introduces a significant technical hurdle: the pose-to-video realism gap. Skeletal coordinates act as an extreme information bottleneck. They discard texture, clothing, facial expressions, and depth. Bridging this gap requires rendering engines to reconstruct high-fidelity human appearances from sparse lines and points without introducing distracting artifacts or falling into the uncanny valley.


Why We Decouple: The Power of Intermediate Poses

The primary motivation for using intermediate skeletal poses is control and tractability. In tasks like Sign Language Production (SLP), mapping text directly to continuous video frames is exceptionally difficult. By breaking the pipeline into two distinct stages, models can first focus entirely on linguistic and structural accuracy.

For instance, Ben Saunders et al. in Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives demonstrate how spoken text can be translated into a latent sign representation and animated via specialized motion "experts" at the frame level. Similarly, Parul Kapoor et al. design a direct pipeline in Towards Automatic Speech to Sign Language Generation that maps raw speech to skeletal keypoint motion.

This structural decoupling is also highly valuable for data compression and privacy. Roshan Prabhakar et al. demonstrate in Reducing latency and bandwidth for video streaming using keypoint extraction and digital puppetry that streaming sparse keypoints instead of raw pixels can bypass low-bandwidth constraints, relying on a generative receiver to reconstruct the video on the fly.


The Realism Bottleneck

The core issue with the decoupled approach is that coordinate-based skeletal sequences are inherently non-photorealistic. When an independent rendering model tries to synthesize video frames from these sparse coordinates, several critical points of failure emerge:


Strategies for Closing the Gap

To overcome these limitations, recent research focuses on enriching the intermediate representation and optimizing how structural data transitions into pixels.

1. Disentangling Appearance from Motion

To synthesize realistic humans, a model must understand what is moving independently of how it is moving. Andreas Blattmann et al. address this in Behavior-Driven Synthesis of Human Dynamics by learning a dedicated representation of dynamics that is entirely independent of the starting posture.

This concept is leveraged for privacy by Ben Saunders et al. in AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation. By explicitly separating the linguistic "pose" from the identity "style," the authors synthesize entirely new, photorealistic avatars to perform the original movements, preserving privacy without losing semantic clarity.

[Text / Speech Input] 
         │
         ▼
[Structural Representation] ───► (Enriched with 3D Meshes or 4D Bone Vectors)
         │
         ├───► [Appearance / Style Matrix] (Identity, Texture, Clothing)
         ▼
[Photorealistic Video Output]

2. Coarse-to-Fine and Multi-Channel Synthesis

Instead of forcing a single network to generate global body movements and intricate finger positions simultaneously, advanced architectures split the task. Tongkai Shi et al. introduce a framework in Pose-Guided Fine-Grained Sign Language Video Generation that decouples motion into coarse structural changes and fine-grained, pose-guided details, significantly stabilizing temporal video quality.

3. Rethinking the Geometric Prior

If 2D joint coordinates discard too much information, the solution is to upgrade the underlying geometry.

4. End-to-End Integration and Continuity

When the pose generator and the video renderer are completely isolated, errors compound. To combat this, Pan Xie et al. propose a more integrated approach in Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation, arguing that tighter coupling between the structural generation and final rendering stages prevents the loss of visual fidelity.

Furthermore, ensuring fluid transitions between discrete poses prevents the final video from looking robotic. Shengeng Tang et al. demonstrate in Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observations how conditional diffusion models can synthesize the complex, non-linear movements that occur between keyframes, providing the temporal smoothness required for natural human video rendering.

The Path Forward

The evolution of this field shows that closing the pose-to-video realism gap is not just about building stronger image generators; it is about designing smarter intermediate representations. By moving from raw 2D coordinates to structured 3D meshes, 4D bone vectors, and disentangled style spaces, researchers provide generative models with the precise geometric constraints they need to produce natural, human-like video while avoiding the uncanny valley.

Key Foundational Work

Adversarial Training for Multi-Channel Sign Language Production (2020) — contributed 2 concept elements to this direction.
Towards Automatic Speech to Sign Language Generation (2021) — contributed 1 concept elements to this direction.
Progressive Transformers for End-to-End Sign Language Production (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pose_to_video_synthesis_7f21728c,
  title = {pose-to-video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pose-to-video-synthesis/rdr_7f21728c9b344c26acce738bd7bce6c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_pose_to_video_synthesis_7f21728c_v1,
  title = {pose-to-video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pose-to-video-synthesis/rdr_7f21728c9b344c26acce738bd7bce6c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake