TL;DR
This paper introduces XNect, a novel real-time system for capturing the 3D motion of multiple people from a single RGB camera. It achieves state-of-the-art accuracy and speed (over 30 fps) on consumer hardware, even in challenging scenarios with occlusions, by employing a unique three-stage pipeline and a new efficient CNN architecture called SelecSLS Net.
What Problem Does This Paper Address?
Capturing the full 3D motion of multiple people in real-time using only a single RGB camera is a significant challenge. Existing methods often struggle with:
- Occlusions: People blocking each other or objects in the scene obscure body parts, making accurate pose estimation difficult.
- Multi-person complexity: Distinguishing and tracking multiple individuals, especially when they are close together or have similar appearances, is computationally intensive.
- Real-time performance: Many advanced 3D pose estimation techniques are too slow for interactive applications, requiring specialized hardware or running offline.
- Output format: Previous methods often output joint positions, which are not directly usable for character animation, unlike joint angles.
This paper aims to bridge this gap by providing a robust, fast, and accurate system that can output full skeletal joint angles for multiple people in real-time from a single RGB feed, making it suitable for applications like virtual reality, gaming, and animation.
Methods
XNect employs a three-stage pipeline:
- Stage I: Per-Frame Pose Inference (SelecSLS Net)
- A novel Convolutional Neural Network (CNN) architecture, SelecSLS Net, is introduced. It uses selective long and short-range concatenation-skip connections instead of additive ones. This design improves information flow, drastically reduces memory footprint, and enables faster inference without sacrificing accuracy compared to standard architectures like ResNet.
- This stage infers:
- 2D joint heatmaps.
- Part Affinity Fields (PAFs) for associating joints to individuals.
- An intermediate 3D pose encoding for visible joints, which only captures local kinematic context (i.e., the pose of a joint relative to its immediate neighbors). This "channel-sparse supervision" is a key insight to avoid encoding conflicts and improve generalization.
- Stage II: 3D Pose Reconstruction
- A lightweight fully-connected network takes the outputs from Stage I (2D joint locations, confidences, and the 3D pose encoding) as input.
- It reconstructs the complete root-relative 3D pose for each individual, including occluded joints, by leveraging global (full-body) context and learned pose priors.
- Stage III: Sequential Motion Capture
- This stage applies space-time skeletal model fitting to the per-frame 2D and 3D predictions.
- It enforces temporal coherence, tracks person identities across frames, and estimates absolute height (optionally using ground plane geometry).
- Crucially, it outputs full skeletal joint angles, which are directly usable for character animation.
Datasets
- Training:
- MPII (single-person 2D pose)
- LSP (single-person 2D pose)
- MS-COCO (multi-person 2D pose)
- MuCo-3DHP (multi-person 3D pose, composited real images)
- Human3.6m (single-person 3D pose, used for retraining Stage II)
- Evaluation:
- MPI-INF-3DHP (single-person 3D pose)
- MuPoTS-3D (multi-person 3D pose)
- 3DPW (multi-person 3D pose in-the-wild)
Key Insights and Ideas
- Selective Skip Connections (SelecSLS Net): Using concatenation-skip connections in a selective manner, rather than dense connections (DenseNet) or additive ones (ResNet), significantly speeds up CNNs while maintaining accuracy. This is crucial for real-time performance.
- Channel-Sparse 3D Pose Encoding: By only supervising the 3D pose encoding for bones directly connected to a visible joint, the system avoids encoding conflicts and improves generalization, especially in challenging scenarios with occlusions and close proximity of individuals. This local context is then combined with global context in Stage II.
- Three-Stage Pipeline for Robustness and Speed: Separating per-frame inference (Stage I & II) from temporal smoothing and animation-ready output (Stage III) allows for efficient processing and robust results. Stage II's ability to leverage 2D pose, confidences, and 3D encoding provides a strong foundation for full 3D reconstruction.
- Joint Angle Output: Providing full skeletal joint angles directly is a significant advantage over methods that output only joint positions, simplifying downstream applications like character animation.
Key Results
- Real-time Performance: Achieves over 30 fps on consumer hardware (e.g., laptop with GTX 1080-MaxQ) for 512x320 input images, with Stage I taking ~21.5 ms.
- Accuracy: State-of-the-art accuracy on challenging multi-person benchmarks like MuPoTS-3D and 3DPW, comparable to or better than many offline methods.
- SelecSLS Net Speedup: The proposed SelecSLS Net architecture is 1.3-1.8x faster than ResNet-50 with comparable accuracy.
- Occlusion Robustness: Demonstrates strong performance in scenes with significant inter-person and person-object occlusions, outperforming prior single-person and some multi-person methods.
- Qualitative Results: Produces temporally coherent and visually plausible 3D motion capture, suitable for driving virtual characters in real-time.
Limitations and Future Work
- Accuracy vs. Multi-view: While robust for a single RGB camera, the accuracy is not yet comparable to multi-view capture systems.
- Neck Visibility Requirement: The system requires the neck joint to be visible for initial person detection, potentially missing individuals if the neck is occluded.
- Close Interactions: Capturing extremely close interactions like hugging is still challenging.
- Identity Tracking: The simple identity tracker can sometimes swap identities during extended occlusions or with similar-looking individuals.
- Pose Dissimilarity: Performance can degrade on poses significantly different from the training data.
Future work includes:
- Improving accuracy, potentially by incorporating physics-based constraints or using more reference joints.
- Developing more sophisticated identity tracking methods.
- Extending the system to handle poses outside the training distribution, possibly through unsupervised or semi-supervised refinement.
- Exploring further architectural optimizations for even faster inference or deployment on specialized hardware.
Summary updated