TL;DR
This paper introduces YeLan, a novel system for high-frequency 3D human pose estimation using event cameras, specifically designed for dynamic dancing environments. By leveraging the unique properties of event cameras and introducing a new physics-aware simulator and dataset, YeLan achieves robust performance in challenging conditions like low lighting and dynamic backgrounds, outperforming existing methods.
What Problem Does This Paper Address?
Technology-Mediated Dancing (TMD) is a growing field, encompassing everything from VR dance games to remote fitness applications. A critical component of these systems is accurate and real-time human pose estimation (HPE). Existing HPE solutions, primarily relying on RGB or RGB-Depth cameras, struggle with dynamic environments. RGB cameras falter in low-light conditions due to motion blur, while depth cameras are power-hungry, have limited range, and are sensitive to clothing color. Furthermore, these traditional cameras capture static backgrounds, which can interfere with pose estimation in dynamic settings. The paper addresses the need for a robust, low-latency, energy-efficient HPE system that can handle the complexities of real-world dancing, including challenging lighting and dynamic backgrounds.
Methods
The proposed system, YeLan, utilizes event cameras, which are inspired by biological vision and excel in low-light and high-dynamic-range scenarios. YeLan consists of two main stages:
- Event Preprocessing and TORE Volume: Event camera data is processed into a modified Time-Ordered Recent Event (TORE) volume representation. This representation preserves both recent and short historical event information, helping to mitigate the "missing torso problem" where static body parts might not generate events. The modification involves a range scaling to ensure better convergence.
- Event Filtering with Human Body Mask Prediction Network: To filter out noise and background events, a U-Net-based network predicts a human body mask. This network can predict future masks, enabling an "early-exit" mechanism that skips further processing if a high confidence score is achieved, saving computational resources.
- Human Pose Estimation Network: This stage employs a ResNet-based feature extractor, a Bidirectional Convolutional LSTM (BiConvLSTM) for spatio-temporal prediction, and an HPE backbone with hourglass-like blocks. It outputs heatmaps for joint projections on three orthogonal planes, from which 3D coordinates are triangulated.
Datasets
The paper introduces two new datasets:
- Synthetic Dataset: Generated using a comprehensive motion-to-event simulator. This simulator is physics-aware and highly customizable, allowing for the generation of event streams with paired pixel-level human masks and 3D joint labels. It includes variations in human models, motion, lighting, background, and camera views.
- Real-World Dataset: Collected from a human subject study in an indoor laboratory setting. Participants performed dances from "Just Dance 2021" under controlled dynamic and static background, as well as low and high-lighting conditions. Ground truth 3D kinematics were captured using a 9-camera motion capture system.
Key Insights and Ideas
- Event Cameras for Dance: The paper demonstrates the significant potential of event cameras for high-frequency 3D HPE in dynamic scenarios like dancing, overcoming limitations of traditional sensors.
- TORE Volume for Missing Torso: The modified TORE volume representation effectively addresses the "missing torso problem" by retaining historical event data, crucial for complete pose estimation.
- Physics-Aware Simulation: The development of a physics-aware simulator is a key enabler for generating realistic and diverse synthetic data, crucial for training robust HPE models.
- Early-Exit Masking for Efficiency: The early-exit mask prediction network not only filters noise but also significantly improves computational efficiency by allowing the pipeline to bypass unnecessary stages.
- Pretraining on Synthetic Data: Pretraining the model on the generated synthetic data significantly improves performance on real-world data, highlighting the value of synthetic data for bridging the gap between simulation and reality.
Key Results
- Performance Improvement: YeLan outperforms baseline methods (Scarpellini et al. [74] and Baldwin et al. [16]) on both synthetic and real-world datasets.
- On the synthetic dataset, YeLan achieved an MPJPE of 46.57, a significant improvement over baselines (91.88 and 59.34).
- On the real-world dataset, YeLan achieved an MPJPE of 96.61, again outperforming baselines (111.57 and 101.967).
- Robustness: YeLan demonstrates strong robustness to various challenging conditions, including low-lighting, dynamic backgrounds, clothing variations, viewing angles, and occlusion.
- Efficiency: The early-exit mask prediction mechanism contributes to computational efficiency, with a balanced accuracy and time consumption achieved at a confidence score threshold of 0.99.
- Comparison with Modalities:
- RGB Cameras: YeLan significantly outperforms RGB cameras in low-lighting conditions, where RGB struggles with motion blur and low SNR.
- RGB-Depth Cameras: Depth cameras are power-hungry, have limited FOV and working distance, and are susceptible to IR interference. YeLan offers a more flexible and efficient alternative.
Limitations and Future Work
- Stationary Camera Assumption: The current YeLan framework is primarily tested with a stationary event camera. Future work aims to extend its transferability to scenarios where the camera is mounted on a moving platform like a car or drone.
- Single-Person Scenario: The current system is designed for single-person scenarios and does not support multiplayer dancing games. Future work will focus on incorporating multi-person segmentation and masking.
- Energy Efficiency: While event cameras are inherently energy-efficient, further exploration into energy-efficient implementation on neuromorphic platforms like Intel Loihi 2 is planned.
Summary updated