Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video

Jixuan He, Chieh Hubert Lin, Lu Qi, Ming-Hsuan Yang

Introduction

Restage4D is a novel pipeline that reanimates deformable 3D objects from a single video by combining the creative flexibility of generative AI with the physical grounding of real-world footage. By using a "video-rewinding" joint training strategy, it leverages the original video to correct geometric artifacts and physical inconsistencies common in AI-generated motion.

Restage4D Concept

The Challenge of 4D Reconstruction from Generative Video

We are currently in a golden age of video generation. Models like Sora can take a text prompt and animate a static scene—making a cow graze or a car turn. However, while these videos look visually appealing, they often lack "physical common sense," resulting in a lack of geometric consistency.

When attempting 4D reconstruction (3D geometry over time) from these synthetic videos, the results often "melt": limbs might swap during an intersection, textures might flicker, or objects might lose their shape when they move behind an obstacle (occlusion). The core challenge is: How can we use the creative power of generative models to "restage" a scene without losing the structural integrity of the real world?

Methodology: Anchoring AI with Real-World Physics

The authors introduce Restage4D, which treats the reconstructed object as a "puppet." The original video provides the skeletal constraints (skinning weights), while the synthetic video provides the new motion (the strings).

Restage4D Pipeline

Key technical innovations include:

  • Video-Rewinding Joint Training: To bridge the gap between the real "base" video and the synthetic "driving" video, the authors play the base video in reverse (rewind) and concatenate it with the driving video. Since both share the same starting frame, this creates a continuous temporal sequence that allows the model to learn a shared 3D representation across both real and fake data.
  • Low-Rank Decomposed 4D Gaussian Splatting: The scene is represented using dynamic 3D Gaussians. Motion is modeled using a small set of "motion bases" that are linearly combined, helping maintain a smooth, articulated structure rather than a chaotic cloud of points.
  • Occlusion-Aware Rigidity (o-ARAP): Generative models often fail when parts of an object are hidden. Restage4D uses an "As-Rigid-As-Possible" (ARAP) loss tuned to be "occlusion-aware." It enforces structural stiffness on parts of the object that are invisible in the synthetic video by "remembering" their geometry from the real video.
  • Disocclusion Backtracing: When new motion reveals a part of the object never seen in the original video (disocclusion), the system "backtraces" those new points into the original 3D canonical space to ensure they stay aligned with the rest of the body.

Key Insights

  • The Real World as a Supervisor: The most striking idea is that the original, unedited video acts as a "physical anchor." Even if the generative AI makes a mistake (like accidentally fusing a road sign to a passing bus), the model uses the "correct" observations from the base video to override the AI's hallucination.
  • Temporal Invariance: 3D reconstruction doesn't inherently care about the arrow of time. By "rewinding" the video, the authors created a mathematically elegant way to supervise a synthetic future with a real past.
  • Selective Regularization: Instead of forcing the whole object to be rigid (which makes motion look stiff), the model only applies heavy structural constraints where the visual evidence is weak, such as in occluded areas.

Datasets and Results

The model was validated on standard benchmarks like DAVIS (for video object segmentation and tracking), PointOdyssey (for long-term 3D point tracking), Neural3DV (for multi-view ground truth), and diverse in-the-wild internet videos.

Restage4D demonstrates significant improvements in both visual quality and physical accuracy:

  • Quantitative Gains: On the DAVIS dataset, Restage4D achieved a PSNR of 27.12, outperforming naive reconstruction baselines. More importantly, it showed a massive jump in Volume Consistency (10.32 vs. 7.41), proving the object's 3D mass stays stable during motion.
  • Error Correction: Qualitatively, the model successfully fixed hallucinations from generative models. For example, in a video of a camel, the generative model erroneously fused a background bush to the camel's hump; Restage4D's joint training successfully "peeled" the bush off and restored the camel's true shape.
  • Tracking Accuracy: On PointOdyssey, the 3D tracking loss was reduced to 0.05, compared to 0.09 when the rigidity and backtracing mechanisms were removed.

Limitations and Future Work

While powerful, Restage4D has its bounds. If the driving video is too distorted or contains extreme "geometry-breaking" artifacts, the real-world prior might not be enough to save the reconstruction. Furthermore, the method assumes the object remains temporally consistent, meaning it might struggle with highly fluid objects (like smoke) or objects with very little texture where tracking becomes impossible. Finally, as with all generative tech, the authors note the risk of creating misleading human-centric content and suggest that prompt filtering is a necessary safeguard for future deployments.

Summary updated

Abstract

Motion is one of the key components in deformable 3D scenes. Generative video models allow users to animate static scenes with text prompts for novel motion, but when it comes to 4D reconstruction, such reanimations often fall apart. The generated videos often suffer from geometric artifacts, implausible motion, and occlusions, which hinder physically consistent 4D reanimation. In this work, we introduce \textbf{Restage4D}, a geometry-preserving pipeline for deformable scene reconstruction from a single edited video. Our key insight is to leverage the unedited original video as an additional source of supervision, allowing the model to propagate accurate structure into occluded and disoccluded regions. To achieve this, we propose a video-rewinding training scheme that temporally bridges the edited and original sequences via a shared motion representation. We further introduce an occlusion-aware ARAP regularization to preserve local rigidity, and a disocclusion backtracing mechanism that supplements missing geometry in the canonical space. Together, these components enable robust reconstruction even when the edited input contains hallucinated content or inconsistent motion. We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, bridging the gap between flexible video synthesis and physically grounded 4D reconstruction.
BibTeX
@inproceedings{he2025restage,
  author = {Jixuan He and Chieh Hubert Lin and Lu Qi and Ming-Hsuan Yang},
  title = {{Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=e50L5Mx93u},
}
Figure