Abstract
Creating deformable 3D content has gained increasing attention with the rise of text-to-image and image-to-video generative models. While these models provide rich semantic priors for appearance, they struggle to capture the physical realism and motion dynamics needed for authentic 4D scene synthesis. In contrast, real-world videos can provide physically grounded geometry and articulation cues that are difficult to hallucinate. One question is raised: Can we generate physically consistent 4D content by leveraging the motion priors of the real-world video? In this work, we explore the task of reanimating deformable 3D scenes from a single video, using the original sequence as a supervisory signal to correct artifacts from synthetic motion. We introduce Restage4D, a geometry-preserving pipeline for video-conditioned 4D restaging. Our approach uses a video-rewinding training strategy to temporally bridge a real base video and a synthetic driving video via a shared motion representation. We further incorporate an occlusion-aware rigidity loss and a disocclusion backtracing mechanism to improve structural and geometry consistency under challenging motion. We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, revealing the potential of video prior in 4D restaging task.
Introduction
The advent of generative video models like Sora and Runway has revolutionized our ability to "hallucinate" complex motions from static images or text prompts. However, these models often operate without a fundamental understanding of 3D physics. This lack of grounding leads to "geometric drift"—phenomena where objects merge into their surroundings, limbs swap positions, or structural integrity dissolves during movement.
Conversely, traditional 4D reconstruction—capturing the 3D shape of an object as it evolves over time—typically requires multi-camera setups or highly constrained movements to maintain accuracy. Restage4D bridges this gap by introducing the task of "4D restaging." This framework reanimates a deformable 3D object from a single real-world video by anchoring synthetic, AI-generated motions to the physical constraints of the original footage. By leveraging the real world as a physical anchor, Restage4D produces 4D reconstructions that are significantly more stable and physically consistent than those generated by video models alone.
Methodology
Restage4D is built upon 4D Gaussian Splatting (4DGS), utilizing a low-rank decomposed representation. The scene is modeled as a collection of 3D points (Gaussians) driven by shared motion bases and coefficients. This effectively treats the object as a digital puppet, where the motion coefficients act similarly to "skinning weights" in traditional animation.
To ensure the model remains faithful to both the real-world geometry and the new synthetic motion, the pipeline introduces three core innovations:
1. Video-Rewinding Training
A primary challenge in 4D restaging is bridging the gap between the "base" (real) video and the "driving" (synthetic) video. Restage4D employs a temporal loop strategy: the base video is played in reverse (rewound) and concatenated with the driving video. Because both sequences share the same starting frame, this creates a continuous temporal bridge. This allows the model to learn a single, consistent 3D representation that spans both real-world observations and imagined dynamics.
2. Occlusion-Aware Rigidity Loss
When synthetic motion causes parts of an object to become hidden from the virtual camera, the model can lose track of the underlying structure. Restage4D implements a k-nearest neighbor (k-NN) graph-based loss. This ensures that nearby points move together rigidly, preserving the object's structural integrity even when specific parts are occluded.
3. Disocclusion Backtracing
When new motions reveal parts of an object that were never visible in the original video, the system "backtraces" these newly visible points into the canonical 3D space. This mechanism ensures that disoccluded geometry is integrated into the master 3D model rather than appearing as disconnected or floating artifacts.
Experimental Validation
The authors evaluated Restage4D across several challenging benchmarks:
- DAVIS: Used to test reanimation on 20 diverse real-world sequences, including animals and vehicles.
- PointOdyssey: A synthetic dataset for long-term 3D tracking, used to quantitatively measure trajectory preservation.
- In-the-wild Videos: Unconstrained internet clips used to demonstrate the model's generalization capabilities.
Key Results
Restage4D demonstrated significant improvements over baseline reconstruction methods:
- Volume Consistency: The model improved volume consistency by nearly 40% compared to naive reconstruction techniques.
- 3D Tracking Accuracy: On the PointOdyssey dataset, Restage4D achieved a 3D tracking loss of 0.05, nearly doubling the accuracy of versions without rigidity constraints (0.09).
- Error Correction: Qualitatively, the model successfully "un-fused" objects and corrected limb-swapping errors present in the raw generative video outputs, effectively using the 3D prior to override the generative model's physical mistakes.
Discussion and Future Work
Restage4D highlights the power of using real-world supervision as a corrective force for generative AI. By treating 3D reconstruction as temporally invariant—where the shape remains the same whether viewed forward or backward—the "rewinding" strategy provides a robust training signal.
However, limitations remain. If the generative driving video is excessively distorted (e.g., the object completely dissolves), the physical prior may not be strong enough to recover the geometry. Additionally, the current method assumes relatively consistent appearance, which may pose challenges for fluid or "shapeshifting" objects like smoke or splashing liquids. Future research will focus on incorporating more complex foundation motion priors and extending the framework to handle multi-object interactions in real-time environments.
Summary updated