Geometric Anchoring for Consistent Video Synthesis

90 concepts73 papers1 proposal

Generative video models have achieved remarkable success in creating fluid, visually stunning sequences from text prompts or single images. However, beneath their photorealistic surfaces lies a fundamental flaw: a lack of spatial permanence. Because traditional video diffusion models and generative adversarial networks (GANs) operate primarily within 2D pixel spaces or unconstrained latent trajectories, they suffer from "digital amnesia." When a virtual camera pans away from an object and returns, the object often morphs or vanishes. Textures crawl across surfaces like liquid, and complex camera movements cause walls to bend and perspective to warp.

The root of this problem is that 2D pixel movements are merely projections of complex 3D interactions, as noted by XIAOJUAN QI et al. in 3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis. Purely temporal architectures, such as sliding-window attention mechanisms or recurrent networks, lack the physical constraints required to preserve structural integrity over long time horizons, leading to severe blurring and geometric drift (Jiajun Wu et al. in Video Extrapolation in Space and Time).

To solve this, a powerful paradigm has emerged: Geometric Anchoring. Instead of forcing a generative model to invent and maintain 3D structure implicitly, researchers are lifting scenes into explicit 3D or 4D geometric proxies—such as point clouds, meshes, or Gaussian primitives—and using them as a rigid scaffold to guide the generative process.


The Core Mechanism: 3D Scaffolding

The intuition behind geometric anchoring is simple: shift the generative model's responsibility from inventing structure to shading and refining details.

[Input View / Prompt] ──> [Lift to 3D Proxy (Point Cloud/Mesh)] ──> [Project to Target View] ──> [Generative Refinement (Diffusion/GAN)] ──> [Consistent Video Frame]

By establishing a 3D proxy before generating frames, the model anchors its predictions to fixed spatial coordinates. For example, Chenjie Cao et al. demonstrate in Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation that transforming a single 2D image into a 3D point cloud via monocular depth estimation, and then re-rendering that point cloud from target camera perspectives, provides an explicit geometric prior. This allows video diffusion models to maintain structural consistency during complex camera trajectories; the model merely needs to fill in occluded areas based on the underlying 3D structure rather than guessing the entire scene layout.

This explicit scaffolding is fundamentally more reliable than implicit depth estimation. As Fenglin Liu et al. show in Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing, utilizing explicit point cloud renderings as a structural guide prevents spatial distortion during aggressive camera rotations and zooms. The 3D proxy acts as a concrete reference that stabilizes the generation of new objects or edits, ensuring they remain locked in place.


Eliminating Flickering via Unified Volumetric Projections

In traditional 2D video editing or frame interpolation, temporal flickering occurs because the model processes frames sequentially or within localized windows, leading to slight variations in pixel values between frames. Geometric anchoring eliminates this problem by treating video generation as a single, unified projection from a global 3D model.

In an early exploration of this concept, Chanya Lueangwattana et al. demonstrated in Removing fences from sweep motion videos using global 3D reconstruction and fence-aware light field rendering that performing a global 3D reconstruction before video restoration eliminates temporal shimmering. Because a single global solution is projected onto the video timeline, frame coherence is guaranteed by design.

Modern architectures achieve this by integrating advanced 3D representations like 3D Gaussian Splatting (3DGS) directly into the generative loop. Zehao Chen et al. leverage this in EvHDR-GS: Event-guided HDR Video Reconstruction with 3D Gaussian Splatting, showing that modeling video as a series of projections from a static 3D scene structurally eliminates flickering. Similarly, Xiaohui Shen et al. introduce a plug-and-play refiner in Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting, lifting 2D edited videos into an optimized 3DGS representation to enforce strict geometric stability across time.


Binding Features to Coordinate Spaces

To make a 2D diffusion model truly respect a 3D scaffold, the generative features and noise must be bound directly to the geometry. If noise is sampled independently per frame, the model will still generate texture crawling.

Shengqu Cai et al. address this in Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models by anchoring generative features and noise to a 3D coordinate system, specifically a UV space. This ensures that visual details remain fixed to specific parts of an object's geometry even as it moves or the camera rotates.

Taking this a step further, Zekai Gu et al. introduce the "Diffusion as Shader" (DaS) framework in Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control. By using "3D tracking videos"—where pixel colors represent persistent 3D coordinates—as a control signal, the model anchors textures to specific geometric points across frames. This mechanism maintains visual identity and eliminates texture sliding even under complex deformations or temporary occlusions.

Traditional Video Diffusion:
[Frame t Noise] ──> [2D Conv/Attn] ──> Texture slides across moving surfaces

Geometric Anchored Diffusion (DaS / Generative Rendering):
[3D Coordinate / UV Map] ──> [Anchor Noise/Features to Geometry] ──> Texture locked to surface

Overcoming Digital Amnesia in World Models

For long-form video generation and autonomous driving world models, sliding-window temporal attention is insufficient because it cannot handle loops or re-visitation. When the camera returns to a previously seen location, the model must remember the exact layout.

Jisang Han demonstrates the power of explicit spatial memory in 3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation. By integrating 3D scene memory into video diffusion models, the framework significantly improves multi-view alignment and reduces geometric drift compared to traditional sliding-window methods. This proves that explicit 3D representations are inherently superior for long-term scene persistence.

This concept is echoed in the work of Arun Mallya et al. (World-Consistent Video-to-Video Synthesis), where all previously generated frames are condensed into a persistent 3D point cloud. When the target camera view is rendered, it projects the current state of this textured point cloud as a guidance image, providing a physically-grounded reference that enforces long-term geometric coherence.

Similarly, Yunpeng Bai shows in GeoVideo: Introducing Geometric Regularization into Video Generation Model that forcing a model to predict depth maps alongside RGB frames and applying a global point-cloud reprojection loss eliminates jitter and geometric drift. The constraints force the model to move beyond superficial 2D pixel patterns toward a functional, stable understanding of 3D space.


Why It Works: A Shift in Perspective

Geometric anchoring works because it resolves the fundamental ambiguity of the 2D-to-3D mapping problem. By providing a low-fidelity or sparse geometric foundation—whether it is a global point cloud derived from sparse views (Fangfu Liu et al. in ReconX), a time-consistent 3D mesh proxy (Yuhao LIU et al. in Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy), or deep features aggregated directly on a 3D surface (Gernot Riegler in Stable View Synthesis)—the generative network no longer has to guess the laws of perspective.

The underlying geometry handles the macro-level physics of motion, parallax, and viewpoint transformations. The generative model can then focus entirely on what it does best: hallucinating high-frequency textures, realistic lighting, and complex occlusions over a structurally unshakeable foundation.

Key Foundational Work

3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis (2019) — contributed 2 concept elements to this direction.
Removing fences from sweep motion videos using global 3D reconstruction and fence-aware light field rendering (2019) — contributed 2 concept elements to this direction.
Deferred Neural Rendering: Image Synthesis using Neural Textures (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometry_guided_video_synthesis_46918da0,
  title = {geometry-guided video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-guided-video-synthesis/rdr_46918da0499b402083c19bab50781471},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometry_guided_video_synthesis_46918da0_v1,
  title = {geometry-guided video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-guided-video-synthesis/rdr_46918da0499b402083c19bab50781471/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake