Abstract
Reconstructing high-quality 3D dynamic humans from monocular video is a significant challenge, especially during rapid motions that induce motion blur. We present ExFMan, a neural rendering framework that leverages hybrid data from standard RGB cameras and high-temporal-resolution event cameras. By introducing an event-oriented 3D velocity field to guide optimization, ExFMan effectively handles motion blur and recovers sharp details. Our method achieves a 3.06 dB PSNR improvement over state-of-the-art methods on both synthetic and real-world datasets.
Introduction
The creation of 3D "digital twins" from monocular video is a cornerstone for applications in VR, AR, and telepresence. While Neural Radiance Fields (NeRF) have demonstrated impressive results for static subjects, they struggle with fast movements like dancing or running. These rapid motions cause significant motion blur in standard RGB frames, leading to "ghostly" artifacts and loss of detail in reconstructed models.
Existing deblurring techniques often fail because they lack an understanding of the underlying 3D human structure. ExFMan addresses this by fusing standard frames with event data—which records intensity changes at microsecond scales—and using a 3D velocity field to model motion uncertainty.
Methodology
ExFMan builds upon the HumanNeRF architecture, introducing several key innovations to handle high-velocity motion:
- Event-Oriented Velocity Field: The framework computes a 3D velocity field in a canonical "T-pose" space. By taking the time derivative of the deformation mapping, the model estimates the speed of every point on the human body, projecting this onto the surface to pinpoint potential blur regions.
- Velocity-Aware Photometric Loss: This loss function treats estimated velocity as a measure of uncertainty. In high-velocity regions where blur is prevalent, the weight of the RGB loss is reduced, preventing the model from overfitting to smeared pixels.
- Velocity-Relative Event Loss: Event cameras provide high-speed supervision in regions where RGB data is unreliable. ExFMan uses this data to "anchor" fast-moving parts to more stable regions, recovering sharp textures.
- Velocity-based Alpha Loss: To maintain sharp boundaries, the model adaptively filters the human mask based on motion, ignoring the haze created by rapid movement.
Key Insights
- Motion as Uncertainty: By modeling color rendering as a Gaussian distribution where velocity correlates with variance, the system naturally prioritizes clear pixels over blurry ones.
- Hybrid Data Synergy: RGB frames provide essential color and texture, while event streams provide the precise temporal "when" and spatial "where" of motion.
- 3D Human Priors: Calculating velocity in canonical space using the SMPL model makes motion estimation far more robust than traditional 2D optical flow methods.
Experimental Results
The researchers evaluated ExFMan using the ZJU-MoCap dataset and a custom real-world dataset captured with a DAVIS346 color event camera.
- Quantitative Performance: ExFMan achieved a 3.06 dB PSNR improvement over existing state-of-the-art methods on the ZJU-MoCap benchmark.
- Qualitative Quality: Visual results demonstrate that ExFMan can reconstruct fine details, such as individual fingers and sharp limb boundaries, even during rapid spins or kicks, where competing methods produce indistinct blobs.
Conclusion and Future Work
ExFMan demonstrates the power of hybrid sensing for 3D human reconstruction. While it significantly improves quality, it introduces higher computational overhead. Future research will explore more efficient representations, such as 3D Gaussian Splatting (3DGS), and more direct integration of event data into initial pose estimation to further enhance robustness and speed.
Summary updated