XNect: real-time multi-person 3D motion capture with a single RGB camera

Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, Christian TheobaltACM Trans. Graphics 2020 · OpenReview

TL;DR

This paper introduces XNect, a novel real-time system for capturing the 3D motion of multiple people from a single RGB camera. It achieves state-of-the-art accuracy and speed (over 30 fps) on consumer hardware, even in challenging scenarios with occlusions, by employing a unique three-stage pipeline and a new efficient CNN architecture called SelecSLS Net.

What Problem Does This Paper Address?

Capturing the full 3D motion of multiple people in real-time using only a single RGB camera is a significant challenge. Existing methods often struggle with:

  • Occlusions: People blocking each other or objects in the scene obscure body parts, making accurate pose estimation difficult.
  • Multi-person complexity: Distinguishing and tracking multiple individuals, especially when they are close together or have similar appearances, is computationally intensive.
  • Real-time performance: Many advanced 3D pose estimation techniques are too slow for interactive applications, requiring specialized hardware or running offline.
  • Output format: Previous methods often output joint positions, which are not directly usable for character animation, unlike joint angles.

This paper aims to bridge this gap by providing a robust, fast, and accurate system that can output full skeletal joint angles for multiple people in real-time from a single RGB feed, making it suitable for applications like virtual reality, gaming, and animation.

Methods

XNect employs a three-stage pipeline:

  1. Stage I: Per-Frame Pose Inference (SelecSLS Net)
    • A novel Convolutional Neural Network (CNN) architecture, SelecSLS Net, is introduced. It uses selective long and short-range concatenation-skip connections instead of additive ones. This design improves information flow, drastically reduces memory footprint, and enables faster inference without sacrificing accuracy compared to standard architectures like ResNet.
  2. This stage infers:
  3. 2D joint heatmaps.
  4. Part Affinity Fields (PAFs) for associating joints to individuals.
  5. An intermediate 3D pose encoding for visible joints, which only captures local kinematic context (i.e., the pose of a joint relative to its immediate neighbors). This "channel-sparse supervision" is a key insight to avoid encoding conflicts and improve generalization.
  6. Stage II: 3D Pose Reconstruction
    • A lightweight fully-connected network takes the outputs from Stage I (2D joint locations, confidences, and the 3D pose encoding) as input.
  7. It reconstructs the complete root-relative 3D pose for each individual, including occluded joints, by leveraging global (full-body) context and learned pose priors.
  8. Stage III: Sequential Motion Capture
    • This stage applies space-time skeletal model fitting to the per-frame 2D and 3D predictions.
  9. It enforces temporal coherence, tracks person identities across frames, and estimates absolute height (optionally using ground plane geometry).
  10. Crucially, it outputs full skeletal joint angles, which are directly usable for character animation.

Datasets

  • Training:
    • MPII (single-person 2D pose)
    • LSP (single-person 2D pose)
    • MS-COCO (multi-person 2D pose)
    • MuCo-3DHP (multi-person 3D pose, composited real images)
    • Human3.6m (single-person 3D pose, used for retraining Stage II)
  • Evaluation:
    • MPI-INF-3DHP (single-person 3D pose)
    • MuPoTS-3D (multi-person 3D pose)
    • 3DPW (multi-person 3D pose in-the-wild)

Key Insights and Ideas

  • Selective Skip Connections (SelecSLS Net): Using concatenation-skip connections in a selective manner, rather than dense connections (DenseNet) or additive ones (ResNet), significantly speeds up CNNs while maintaining accuracy. This is crucial for real-time performance.
  • Channel-Sparse 3D Pose Encoding: By only supervising the 3D pose encoding for bones directly connected to a visible joint, the system avoids encoding conflicts and improves generalization, especially in challenging scenarios with occlusions and close proximity of individuals. This local context is then combined with global context in Stage II.
  • Three-Stage Pipeline for Robustness and Speed: Separating per-frame inference (Stage I & II) from temporal smoothing and animation-ready output (Stage III) allows for efficient processing and robust results. Stage II's ability to leverage 2D pose, confidences, and 3D encoding provides a strong foundation for full 3D reconstruction.
  • Joint Angle Output: Providing full skeletal joint angles directly is a significant advantage over methods that output only joint positions, simplifying downstream applications like character animation.

Key Results

  • Real-time Performance: Achieves over 30 fps on consumer hardware (e.g., laptop with GTX 1080-MaxQ) for 512x320 input images, with Stage I taking ~21.5 ms.
  • Accuracy: State-of-the-art accuracy on challenging multi-person benchmarks like MuPoTS-3D and 3DPW, comparable to or better than many offline methods.
  • SelecSLS Net Speedup: The proposed SelecSLS Net architecture is 1.3-1.8x faster than ResNet-50 with comparable accuracy.
  • Occlusion Robustness: Demonstrates strong performance in scenes with significant inter-person and person-object occlusions, outperforming prior single-person and some multi-person methods.
  • Qualitative Results: Produces temporally coherent and visually plausible 3D motion capture, suitable for driving virtual characters in real-time.

Limitations and Future Work

  • Accuracy vs. Multi-view: While robust for a single RGB camera, the accuracy is not yet comparable to multi-view capture systems.
  • Neck Visibility Requirement: The system requires the neck joint to be visible for initial person detection, potentially missing individuals if the neck is occluded.
  • Close Interactions: Capturing extremely close interactions like hugging is still challenging.
  • Identity Tracking: The simple identity tracker can sometimes swap identities during extended occlusions or with similar-looking individuals.
  • Pose Dissimilarity: Performance can degrade on poses significantly different from the training data.

Future work includes:

  • Improving accuracy, potentially by incorporating physics-based constraints or using more reference joints.
  • Developing more sophisticated identity tracking methods.
  • Extending the system to handle poses outside the training distribution, possibly through unsupervised or semi-supervised refinement.
  • Exploring further architectural optimizations for even faster inference or deployment on specialized hardware.

Summary updated

Abstract

We present a real-time approach for multi-person 3D motion capture at over 30 fps using a single RGB camera. It operates successfully in generic scenes which may contain occlusions by objects and by other people. Our method operates in subsequent stages. The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals. We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy. In the second stage, a fullyconnected neural network turns the possibly partial (on account of occlusion) 2D pose and 3D pose features for each subject into a complete 3D pose estimate per individual. The third stage applies space-time skeletal model fitting to the predicted 2D and 3D pose per subject to further reconcile the 2D and 3D pose, and enforce temporal coherence. Our method returns the full skeletal pose in joint angles for each subject. This is a further key distinction from previous work that do not produce joint angle results of a coherent skeleton in real time for multi-person scenes. The proposed system runs on consumer hardware at a previously unseen speed of more than 30 fps given 512x320 images as input while achieving state-of-the-art accuracy, which we will demonstrate on a range of challenging real-world scenes.
BibTeX
@article{mehta2020xnect,
  author = {Dushyant Mehta and Oleksandr Sotnychenko and Franziska Mueller and Weipeng Xu and Mohamed Elgharib and Pascal Fua and Hans-Peter Seidel and Helge Rhodin and Gerard Pons-Moll and Christian Theobalt},
  title = {{XNect: real-time multi-person 3D motion capture with a single RGB camera}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=OXDJU11xr6A},
}
Figure