PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting

Yijun Xu, Jingrui Zhang, Hongyi Liu, Yuhan Chen, Yuanyang Wang, Qingyao Guo, Dingwen Wang, Lei Yu, Chu HearXiv 2025 · DOI · arXiv

Introduction

Dynamic 3D reconstruction—the process of transforming 2D video into a navigable 3D scene over time—remains a significant challenge in computer vision, with profound implications for VR, gaming, and cinematography. While techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have revolutionized static scene capture, they often fail when confronted with high-speed motion.

The authors identify three primary obstacles:

  1. Motion Blur: Standard RGB cameras suffer from long exposure times, turning fast-moving objects into blurry streaks and losing critical geometric detail.
  2. Modeling Paradigms: Existing dynamic models frequently rely on deformation fields that cannot handle large displacements, leading to mathematical breakdown and visual artifacts.
  3. Physical Inconsistency: Purely visual models lack an understanding of Newtonian physics, often resulting in "disordered drift" where objects jitter or float unrealistically.

The PEGS Framework

To address these issues, the researchers introduced PEGS (Physics-Event enhanced Gaussian Splatting). This framework integrates the high temporal resolution of event cameras with the structural constraints of Newtonian dynamics.

Overview of the PEGS framework, integrating RGB blur frames and event streams with physical priors for 4D sharp reconstruction.

The PEGS pipeline operates in two tightly coupled stages:

1. Target Modeling and Deblurring

The system first "unblurs" the moving object by leveraging Event Cameras. These bio-inspired sensors record brightness changes at microsecond intervals, providing a high-speed temporal skeleton that guides the deblurring of standard RGB frames. Once sharpened, the object is modeled as a collection of 3D Gaussians within a normalized coordinate system, effectively decoupling its appearance from its global trajectory.

2. Motion Recovery via Triple-Level Supervision

To accurately track the object's path through 3D space, PEGS employs a triple-level supervision scheme:

  • Acceleration Consistency: By enforcing Newtonian dynamics (e.g., constant gravity), the model ensures smooth, physically plausible trajectories, preventing unrealistic "teleportation."
  • Event Stream Alignment: The model synthesizes virtual events based on its motion estimates and compares them to real-world event data, providing high-frequency guidance.
  • Kalman Fusion: A Kalman filter fuses physical predictions with visual observations (optical flow) to determine the mathematically optimal trajectory.

Additionally, the authors introduced Motion-aware Simulated Annealing (MSA), which adaptively adjusts the learning rate based on object velocity—using higher rates for fast-moving phases to ensure convergence and lower rates for slower phases to refine fine details.

Datasets and Evaluation

The study contributes a new RGB-Event paired dataset focused on large-scale, "wild" rigid motion, such as projectile motion and free fall. This dataset includes:

  • Synthetic Data: 6 scenes rendered in Blender with simulated events.
  • Real-World Data: 9 scenes captured via a custom hardware rig using a beam splitter to align a FLIR RGB camera and a DVXplorer event camera on the same optical path.

Key Results

PEGS demonstrates a significant leap over state-of-the-art dynamic methods like 4DGS and DynamicGS.

6DoF tracking results demonstrating high accuracy (ATE: 0.080) and reconstruction quality (PSNR: 29.70).
  • Reconstruction Quality: On synthetic benchmarks, PEGS achieved a PSNR of 29.70, avoiding the "spiky" artifacts and blurring seen in competing methods.
  • Tracking Accuracy: The framework reduced the Absolute Trajectory Error (ATE) by factors of 2x to 5x compared to RGB-only approaches.
  • Computational Efficiency: PEGS completes a reconstruction in approximately 56 minutes, a massive improvement over specialized deblurring methods that can take up to 23 hours.

Conclusion and Future Work

By using physics as a regularizer and event streams as "temporal glue," PEGS successfully reconstructs sharp 4D scenes from blurry, high-speed input. While currently limited to rigid objects and constant force fields (like gravity), the authors plan to extend the framework to handle deformable objects and variable force environments, such as wind gusts.

Summary updated

BibTeX
@misc{xu2025pegs,
  author = {Yijun Xu and Jingrui Zhang and Hongyi Liu and Yuhan Chen and Yuanyang Wang and Qingyao Guo and Dingwen Wang and Lei Yu and Chu He},
  title = {{PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting}},
  year = {2025},
  eprint = {2511.17116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.17116},
}
Figure