Matching and Recovering 3D People from Multiple Views

Alejandro Pérez-Yus, Antonio AgudoWACV 2022 · OpenReview

TL;DR

This paper presents a novel approach to simultaneously match and recover 3D human poses from multiple calibrated camera views. It introduces a robust multi-view matching algorithm that leverages an "uncertainty geometric consistency" to group 2D detections, even with occlusions and bad detections, and then efficiently infers 3D poses using a 3D pictorial structure model with physico-geometric constraints. The method achieves state-of-the-art results on challenging multi-person scenarios.

What Problem Does This Paper Address?

Estimating the 3D pose of multiple people in a scene from multiple camera views is a fundamental yet challenging problem in computer vision. While 2D pose estimation has seen significant advancements, accurately inferring 3D poses, especially in crowded and dynamic environments, remains difficult. Key challenges include:

  • Multi-body ambiguity: Distinguishing and tracking individual people when they move, deform, or interact.
  • Occlusions and partial views: People are often partially or fully hidden in some camera views, making it hard to get complete pose information.
  • Noisy detections: 2D pose detectors can produce inaccurate joint locations or even miss detections.
  • Unknown number of people: The algorithm should not require prior knowledge of how many people are present.
  • Computational efficiency: Many existing methods are computationally expensive, limiting their real-world applicability.

This paper aims to address these issues by proposing a unified framework that jointly handles the matching of 2D detections across views and the subsequent 3D pose recovery.

Methods

The proposed approach consists of two main stages: multi-view matching and 3D pose recovery.

  1. 2D Pose Detection: The process begins by applying a standard 2D human pose detector (e.g., Cascaded Pyramid Network) to each input image from multiple calibrated cameras. This yields bounding boxes and 2D joint positions with associated heatmaps for each detected person in every view.
  2. Multi-view Matching: This is the core novelty of the paper.
  3. Affinity Estimation: Instead of just relying on simple geometric consistency, the authors propose an "uncertainty geometric consistency" measure. This involves virtually hallucinating a 3D point from a pair of 2D joint detections in two views and re-projecting it onto the heatmaps of those views. The affinity score is derived from the quality of these re-projections, normalized by the maximum heatmap values.
  4. Enforcing Uncertainty Geometric Consistency: To improve robustness, the method extends this by considering additional views. If a re-projected 3D point falls within the image plane of a third view, its score contributes to the overall affinity, effectively penalizing matches that are inconsistent across multiple views. This is formulated using a novel affinity matrix that incorporates uncertainty.
  5. Robust Multi-view Matching Algorithm: The computed affinities are then fed into an optimization algorithm that aims to find a cycle-consistent matching between bounding boxes across all views. This algorithm is designed to be robust against bad detections and partial observations, and it implicitly solves the 2D body correspondences. The optimization problem is formulated using a nuclear norm for low-rank approximation and an L1-norm for robustness to noise, with additional constraints to ensure valid matching.
  6. 3D Pose Recovery: Once the 2D detections for the same person across different views are successfully matched, the 3D pose is recovered.
  7. 3D Pictorial Structure (3DPS) Model: The authors employ a 3DPS model, which represents the human body as a graph where nodes are joints and edges are body parts.
  8. Physico-Geometric Constraints: To enhance accuracy and realism, the 3DPS model is augmented with physico-geometric constraints. These include:
  9. Data Term: Likelihood based on the 2D heatmaps from the detector.
  10. Bone Length Prior: Penalizing poses where joint distances deviate from learned typical bone lengths.
  11. Joint Angle Prior: Enforcing physically plausible angles between connected body parts.
  12. Heatmap Prior: Penalizing deviations from ideal Gaussian heatmaps, helping to filter out noisy detections.
  13. Inference: The 3D pose is inferred by maximizing the posterior probability defined by the 3DPS model and its priors. To manage computational complexity, especially with many candidate points, the method incorporates an interpolation strategy and a mechanism to discard less likely candidates early in the optimization.

Datasets

The paper evaluates its approach on several datasets:

  • Campus and Shelf: These datasets are used for quantitative evaluation of both multi-body matching and 3D pose estimation. The authors mention manually annotating ground truth for matching evaluation on these datasets.
  • KTH Football II [31]: Used for evaluating performance in challenging scenarios with motion blur, odd postures, and high appearance similarity between players.
  • CMU Panoptic [30]: Used to demonstrate the algorithm's ability to handle multiple people and strong occlusions across many camera views.

The paper also references the Human3.6M dataset [28] as the source for learning the mean and standard deviation of bone lengths used in the physico-geometric constraints.

Key Insights and Ideas

  • Uncertainty Geometric Consistency: The core idea of using re-projection errors across multiple views to define an affinity measure is a significant improvement over pairwise consistency. It allows the system to "vote" for correct matches and penalize inconsistent ones, leading to more robust matching.
  • Joint Matching and Recovery: The paper successfully demonstrates that jointly addressing the matching and recovery stages, rather than treating them as separate sequential problems, leads to better overall performance.
  • Robustness to Imperfect Detections: The proposed affinity measure and optimization framework are explicitly designed to handle noisy 2D detections, occlusions, and partial views, which are common in real-world scenarios.
  • Physico-Geometric Constraints as Priors: Integrating learned bone length and joint angle constraints into the 3D pose recovery significantly improves the realism and accuracy of the reconstructed poses, especially in challenging situations.
  • Efficient Inference with Candidate Interpolation: The strategy of interpolating candidate 3D points and pruning less likely ones during inference is crucial for making the 3D pose recovery computationally tractable for complex scenes.

Key Results

  • Multi-view Matching: The proposed method achieves state-of-the-art results on the Campus and Shelf datasets, outperforming existing methods in precision, recall, and F1-score, particularly in challenging scenarios with occlusions and similar appearances. For example, on the Campus dataset, their F1-score reaches 93.46% (using geometry) and 92.40% (using geometry + ReID), while on Shelf, it's 95.04% and 81.92% respectively, often surpassing [20].
  • 3D Pose Recovery: The method demonstrates superior accuracy in 3D pose estimation compared to baselines on the Campus and Shelf datasets, as measured by PCP5 and MPJPE. For instance, on Campus, their PCP5 is 96.71%, outperforming most other methods. They also show competitive results on KTH Football II and CMU Panoptic, handling complex motions and multiple people effectively.
  • Robustness: Qualitative results (e.g., in Figure 3) highlight the method's ability to correctly match bodies even with significant occlusions (green arrows) and avoid incorrect matches (red arrows) that plague competing methods.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their method is robust, it still relies on the quality of the initial 2D pose detections. Very poor 2D detections can still lead to incorrect matches or poses.
  • The computational cost, while improved through optimization and candidate pruning, might still be a factor for real-time applications with extremely large numbers of people or very high-resolution video.
  • Future Work:
    • The authors express interest in extending their research to perform recognition of human activities. This suggests that the rich pose information and tracking capabilities developed in this work could be a strong foundation for higher-level scene understanding.

Summary updated

Abstract

This paper introduces an approach to simultaneously match and recover 3D people from multiple calibrated cameras. To this end, we present an affinity measure between 2D detections across different views that enforces an uncertainty geometric consistency. This similarity is then exploited by a novel multi-view matching algorithm to cluster the detections, being robust against partial observations as well as bad detections and without assuming any prior about the number of people in the scene. After that, the multi-view correspondences are used in order to efficiently infer the 3D pose of each body by means of a 3D pictorial structure model in combination with physico-geometric constraints. Our algorithm is thoroughly evaluated on challenging scenarios where several human bodies are performing different activities which involve complex motions, producing large occlusions in some views and noisy observations. We outperform state-of-the-art results in terms of matching and 3D reconstruction.
BibTeX
@inproceedings{perezyus2022matching,
  author = {Alejandro Pérez-Yus and Antonio Agudo},
  title = {{Matching and Recovering 3D People from Multiple Views}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=CwsJNUEGDI},
}
Figure