TL;DR
This paper introduces "Flycon," a novel system for real-time, infrastructure-free human pose estimation using a swarm of aerial vehicles (drones). By jointly optimizing drone trajectories and human pose, Flycon overcomes the limitations of traditional motion capture systems, enabling tracking in challenging and dynamic environments.
What Problem Does This Paper Address?
Traditional motion capture systems, crucial for applications like character animation, VR, and biomechanics, typically rely on fixed, accurately calibrated camera setups. This setup is expensive, time-consuming to deploy, and fundamentally limits motion capture to controlled indoor environments. It becomes infeasible for tracking subjects in large-scale, dynamic, or outdoor scenarios, such as following an athlete during a workout, capturing actors in remote locations, or even filming in challenging terrains. The core problem is the lack of a robust, real-time system that can perform accurate human pose estimation without relying on pre-installed infrastructure.
Methods
Flycon tackles this by integrating two key components: a multi-drone system for capturing the subject and a sophisticated estimation algorithm.
- Swarm of Aerial Vehicles: A swarm of camera-equipped drones acts as a dynamic, mobile capture platform. These drones are controlled to maintain optimal viewpoints of the subject, ensuring continuous visibility of markers.
- Joint Optimization Framework: The system employs an "error-state Kalman filter" (ESKF) to jointly estimate the 3D positions of human joints, the poses (position and orientation) of each drone, and the lengths of the human's bones. This is a significant departure from traditional methods that assume static cameras.
- Robo-centric Formulation: To avoid drift in pose estimation over time, the system uses a "robo-centric" approach. Instead of estimating poses relative to a fixed world frame, all poses are expressed relative to a moving reference frame (the lead drone). This significantly reduces linearization errors and improves long-term accuracy.
- Model Predictive Control (MPC): A receding horizon controller is used to compute drone inputs. This MPC optimizes drone trajectories to maximize marker visibility, ensuring the subject remains in view from multiple angles, even during fast movements.
- Active Markers: The system uses active IR-LED markers worn by the subject. These markers are uniquely identified by temporal patterns, allowing for robust detection and labeling of joint positions in the drone camera feeds.
Datasets
Not discussed in the paper. The experiments were conducted in real-world scenarios with subjects performing various actions.
Key Insights and Ideas
- Environment Independence: The most significant insight is achieving robust human pose estimation without any fixed infrastructure. This opens up motion capture to a vast range of previously inaccessible scenarios.
- Joint Optimization is Key: Treating drone poses and human poses as interdependent states, rather than separate problems, is crucial for achieving accurate and stable tracking. The drones' movements directly inform the human pose estimation, and vice-versa.
- Robo-centric Filtering for Stability: The robo-centric ESKF formulation is a clever solution to the problem of drift in long-term pose estimation, a common issue in traditional Kalman filtering.
- MPC for Dynamic Viewpoint Control: Using MPC to actively control the drones ensures that the subject remains visible, even with fast motion, which is critical for maintaining data quality.
Key Results
- Real-time Performance: The system operates in real-time, achieving a frame rate of 30Hz for image capture and processing.
- Accuracy: In a ground-truth comparison with a Vicon system, the method achieved a standard deviation of 2.2cm in estimating the distance between a subject's hand and a drone camera over a 30-second sequence.
- Long-Term Tracking: The system demonstrated successful tracking of a subject over a 3-minute period and a trajectory length of 170 meters, with an observed absolute position drift of approximately 2 meters.
- Challenging Scenarios: The paper showcases successful demonstrations in various challenging environments, including fast motion (jumping jacks), climbing walls, long outdoor trajectories, and indoor settings.
Limitations and Future Work
- Marker Dependency: The current system relies on active IR-LED markers. The authors suggest future work could integrate deep-learning-based markerless pose estimation methods.
- Visibility Loss: For very fast movements, visibility could still be lost with the current two-drone setup. This could be mitigated by using more drones.
- Integration with Deep Learning: The authors envision integrating their framework with deep learning models for direct 3D pose prediction from images, which would require significant algorithmic changes.
- Environmental Features: Extracting and utilizing environmental features could further enhance localization accuracy.
- Dense Surface Reconstruction: Extending the method to capture dense surface deformation via model-fitting is another promising avenue.
- Applications: The authors are keen to explore applications in graphics, AR/VR, and biomechanics.
Summary updated