E-Sem3DGS: Monocular Human and Scene Reconstruction via Event-Aided Semantic 3DGS

Xiaoting Yin, Hao Shi, Kailun Yang, Jia Zhai, Shangwei Guo, Kaiwei WangSensors 2025 · DOI

TL;DR

This paper introduces E-Sem3DGS, a novel framework for reconstructing animatable 3D human avatars and their surrounding scenes from monocular, motion-blurred videos. By integrating event camera data and semantic attributes within a unified 3D Gaussian Splatting representation, E-Sem3DGS effectively mitigates motion blur and achieves state-of-the-art performance in challenging scenarios.

What Problem Does This Paper Address?

Reconstructing realistic, animatable 3D humans and their environments from monocular videos is a significant challenge, especially when dealing with motion blur. Existing methods often struggle with:

  • Motion Blur: Fast-moving subjects in videos create blur, making it difficult to accurately capture geometry and appearance across frames. Standard cameras are susceptible to this.
  • Foreground Focus: Many approaches prioritize human reconstruction, leading to incomplete or inaccurate scene modeling.
  • Scene-Human Separation: Differentiating between the dynamic human and the static background can be difficult, especially under blur, where segmentation masks become unreliable.
  • Computational Cost: Many neural rendering techniques are computationally expensive for both training and inference.

Event cameras, with their high temporal resolution and robustness to motion blur, offer a promising solution. However, effectively integrating their data with traditional intensity-based methods for joint human and scene reconstruction, particularly under motion blur, remains an open problem.

Methods

E-Sem3DGS builds upon the 3D Gaussian Splatting (3DGS) framework and introduces several key innovations:

  • Unified Semantic Representation: Instead of separate representations for humans and scenes, E-Sem3DGS uses a single set of 3D Gaussians, each augmented with a learnable semantic attribute. This attribute "softly" separates human (semantic value 1) from scene (semantic value 0) content, allowing for a unified optimization pipeline.
  • SMPL Initialization: Human Gaussians are initialized using priors from the Skinned Multi-Person Linear (SMPL) model, providing a strong geometric starting point. Scene Gaussians are initialized by sampling a surrounding cube.
  • Event-Aided Optical Flow Supervision: This is a core contribution. The system leverages event camera data to derive optical flow. This event-derived flow is then used to supervise the optical flow calculated between rendered intensity frames. This supervision is selectively applied to regions with high motion, effectively guiding the deblurring process and enforcing temporal coherence where it's most needed.
  • Scene-Color MLP: A dedicated Multi-Layer Perceptron (MLP) is used to model the appearance of static scene elements, offering more flexibility and robustness than traditional spherical harmonics.
  • Joint Optimization: The framework jointly optimizes the 3D Gaussians (geometry, appearance, semantics), human deformation (using SMPL parameters), skinning, and the scene-color MLP. This end-to-end approach ensures consistency between human and scene reconstructions.

Datasets

  • ZJU-MoCap-Blur: This dataset was created by taking sequences from the ZJU-MoCap dataset and applying motion blur using Super-SloMo. It includes synchronized RGB frames and, for simulation purposes, event data derived from optical flow estimation on ground truth images. The authors selected view "1" for reconstruction.
  • MMHPSD-Blur: This dataset, originally designed for 3D human pose estimation, was adapted for this work. It contains event-grayscale image pairs, SMPL ground-truth parameters, and event-based optical flow. Motion blur was also applied to the RGB frames using Super-SloMo.

Key Insights and Ideas

  • Semantic Attributes as a Differentiable Gating Mechanism: The learnable semantic attributes allow the model to dynamically control how Gaussians are deformed. This is crucial for disentangling human motion from static scene elements and allows for self-correction even if initial classifications are imperfect.
  • Event Optical Flow as a Robust Intermediate Representation: Instead of directly using raw event data or relying on potentially noisy 2D deblurring pre-processing, using event-derived optical flow provides a more stable and sensor-agnostic signal for guiding 3D reconstruction and deblurring.
  • Unified Representation Simplifies Optimization: By avoiding separate Gaussian sets for humans and scenes (as in some prior work), E-Sem3DGS simplifies the optimization process and reduces complexity.
  • Targeted Motion Blur Mitigation: Applying optical flow supervision specifically to high-motion regions is more efficient and effective than a blanket deblurring approach, focusing computational effort where it matters most.

Key Results

  • Significant PSNR Improvement: On the ZJU-MoCap-Blur dataset, E-Sem3DGS achieved an average full-frame PSNR of 32.56, a substantial improvement of +49.7% over the baseline 3DGS-Avatar (21.75 PSNR).
  • Strong Performance on MMHPSD-Blur: The method improved PSNR from 25.23 to 28.63 (+13.48%) on the MMHPSD-Blur dataset compared to HUGS.
  • Qualitative Improvements: Visualizations show sharper human contours and more consistent static scene reconstruction, especially in challenging motion-blurred regions, compared to baseline methods.
  • Efficiency: Training takes approximately 35 minutes on a single NVIDIA RTX 3090 Ti GPU for a ~400-frame sequence, and inference achieves ~19 FPS, demonstrating a good balance between quality and speed.

Limitations and Future Work

  • Training Complexity: The use of event-based optical flow supervision adds to the training complexity, although inference remains efficient.
  • Dataset Limitations: Current evaluations are primarily on indoor scenes due to the scarcity of real-world event camera datasets. Outdoor robustness and dynamic lighting conditions are areas for future exploration.
  • Generalization to Unseen Poses: Performance might be suboptimal for poses significantly outside the training distribution.
  • Future Directions:
    • Extending to more complex interactions and multi-person scenarios.
  • Exploring training on larger-scale real event-camera datasets.
  • Investigating robustness in unconstrained outdoor environments with dynamic lighting and complex backgrounds.
  • Integrating generative models (e.g., diffusion models) to augment the pose space and improve generalization to unseen poses.
  • Leveraging large-scale human motion datasets (e.g., AMASS) and zero-shot learning for broader identity generalization.

Summary updated

Abstract

Reconstructing animatable humans, together with their surrounding static environments, from monocular, motion-blurred videos is still challenging for current neural rendering methods. Existing monocular human reconstruction approaches achieve impressive quality and efficiency, but they are designed for clean intensity inputs and mainly focus on the foreground human, leading to degraded performance under motion blur and incomplete scene modeling. Event cameras provide high temporal resolution and robustness to motion blur, making them a natural complement to standard video sensors. We present E-Sem3DGS, a semantically augmented 3D Gaussian Splatting framework that leverages hybrid event-intensity streams to jointly reconstruct explicit 3D volumetric representations of human avatars and static scenes. E-Sem3DGS maintains a single set of 3D Gaussians in Euclidean space, each endowed with a learnable semantic attribute that softly separates dynamic human and static scene content. We initialize human Gaussians from Skinned Multi-Person Linear (SMPL) model priors with semantic values set to 1 and scene Gaussians by sampling a surrounding cube with semantic values set to 0, then jointly optimize geometry, appearance, and semantics. To mitigate motion blur, we derive optical flow from events and use it to supervise image-based optical flow between rendered frames, enforcing temporal coherence in high-motion regions and sharpening both humans and backgrounds. On the motion-blurred ZJU-MoCap-Blur dataset, E-Sem3DGS improves the average full-frame PSNR from 21.75 to 32.56 (+49.7%) over previous methods. On MMHPSD-Blur, our method improves PSNR from 25.23 to 28.63 (+13.48%).
BibTeX
@inproceedings{yin2025e,
  author = {Xiaoting Yin and Hao Shi and Kailun Yang and Jia Zhai and Shangwei Guo and Kaiwei Wang},
  title = {{E-Sem3DGS: Monocular Human and Scene Reconstruction via Event-Aided Semantic 3DGS}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s26010188},
}
Figure