TL;DR
ExFMan is a new neural rendering framework that reconstructs high-quality 3D dynamic humans from blurry, fast-moving monocular videos. By fusing standard RGB frames with high-temporal-resolution data from event cameras and using a novel 3D velocity field to guide the optimization, it achieves a significant 3.06 dB PSNR improvement over existing state-of-the-art methods.
What Problem Does This Paper Address?
Reconstructing a 3D "digital twin" of a moving human from a single camera (monocular video) is a holy grail for VR, AR, and telepresence. While Neural Radiance Fields (NeRF) have made this possible for slow movements, they fall apart when things get fast.
Rapid motions—like dancing, running, or jumping—cause motion blur in standard RGB cameras. This blur creates a fundamental "inconsistency" problem: the same body part looks different across frames because of the smearing effect. Standard models try to average this out, resulting in "ghostly" or hazy reconstructions, especially around the hands and feet. Existing deblurring tools often fail because they don't understand the underlying 3D structure of the human body.
Methods
The authors introduce ExFMan, which stands for Exploiting Frames and Manifold velocity. The framework is built on top of HumanNeRF but introduces three key innovations:
- Event-Oriented Velocity Field: The system calculates a 3D velocity field in the "canonical space" (a standardized T-pose). By calculating the derivative of the deformation mapping over time, the model knows exactly how fast every point on the human body is moving. It then projects this velocity onto the human surface to identify exactly where motion blur is likely to occur.
- Velocity-Aware Photometric Loss: Instead of forcing the model to match a blurry pixel exactly, this loss treats the estimated velocity as "uncertainty." In regions with high velocity (and thus high blur), the model lowers the weight of the RGB loss, preventing it from "learning" the blur.
- Velocity-Relative Event Loss: This is where the "hybrid" part comes in. Event cameras are bio-inspired sensors that only record changes in intensity at microsecond scales, making them virtually immune to motion blur. ExFMan uses this high-speed event data to provide supervision in the fast-moving regions where the RGB data is unreliable.
- Velocity-based Alpha Loss: To ensure the human has sharp boundaries, the model adaptively filters the human mask based on motion, focusing on clear edges and ignoring the "haze" created by rapid movement.
Datasets
The researchers evaluated ExFMan on both synthetic and real-world data:
- Synthetic (ZJU-MoCap Extension): They took the standard ZJU-MoCap dataset and simulated motion blur and event data using the RIFE (interpolation) and V2E (event simulator) algorithms. They focused on six subjects performing complex motions.
- Real-World Data: They captured a new dataset using a DAVIS346 color event camera, which provides hardware-aligned RGB frames and event streams simultaneously at a resolution of 346x260.
Key Insights and Ideas
- Velocity as a Proxy for Uncertainty: One of the "aha" moments is treating motion not just as a physical property, but as a statistical variance. By modeling the color rendering as a Gaussian distribution where velocity equals variance, the math naturally tells the model: "Don't trust the blurry pixels."
- Events are the "High-Frequency" Anchor: While RGB frames provide the "what" (color and texture), events provide the "when" and "where" (precise motion timing). By "anchoring" high-velocity regions to low-velocity ones using event integrals, the model can recover sharp details that are invisible in the raw video frames.
- Canonical vs. Observation Space: By calculating velocity in the canonical (T-pose) space, the model leverages the 3D human prior (the SMPL model), making the velocity estimation much more robust than simple 2D optical flow.
Key Results
ExFMan significantly outperforms both standard baselines and two-stage "deblur-then-reconstruct" pipelines:
- Quantitative: On the ZJU-MoCap dataset, ExFMan achieved a 3.06 dB PSNR improvement over the best-performing comparison methods.
- Qualitative: In visual comparisons, ExFMan successfully reconstructs individual fingers and clear leg boundaries during rapid kicks or spins, whereas other methods produce a single indistinct "blob" of color.
- Robustness: The method remains effective even as motion blur intensity increases (simulated by longer exposure times).
Limitations and Future Work
- Computational Overhead: Estimating the 3D velocity field and processing event data increases the computational and memory costs compared to standard NeRF approaches.
- Reliance on Priors: The quality of the reconstruction still depends on the accuracy of the initial pose estimation and human masks. If the initial pose is wildly wrong due to extreme blur, the velocity field will be inaccurate.
- Future Directions: The authors suggest moving toward more efficient representations like 3D Gaussian Splatting (3DGS) to speed up rendering, and developing more robust pose estimation methods that utilize event data directly from the start.
Summary updated