Semantically Synchronizing Multiple-Camera Systems with Human Pose Estimation

Zhe Zhang, Chunyu Wang, Wenhu Qin

TL;DR

This paper introduces a novel, training-free framework to synchronize multiple low-cost cameras by leveraging semantic human pose estimation. By treating human poses as a common signal across views, the method achieves robust sub-frame temporal synchronization, even in challenging scenarios with occlusion and noisy pose data.

What Problem Does This Paper Address?

Modern computer vision applications, such as 3D human pose estimation and body reconstruction, often benefit from multiple camera views to overcome occlusion and expand coverage. While high-end, professionally synchronized camera systems exist, they are prohibitively expensive for large-scale commercial applications like smart stores. Budget-friendly camera setups, however, suffer from temporal synchronization issues due to inherent clock drift. Traditional synchronization methods rely on external signals (e.g., hardware clocks, pattern boards, flashes) or manual calibration, which are either costly, cumbersome, or limited in their ability to handle significant temporal displacements. This paper addresses the need for an affordable, robust, and automated solution to temporally synchronize multiple low-cost cameras using only the visual information from the cameras themselves.

Methods

The proposed framework synchronizes multiple cameras in two main stages:

  1. Two-View Synchronization:
    • Pose Estimation: An off-the-shelf 2D human pose estimator (HigherHRNet) is used to extract human pose keypoints from each camera's video feed. Crucially, the pose estimator is not fine-tuned, making the method adaptable to various systems.
  2. Epipolar Geometry and Distance: The paper leverages epipolar geometry to define a distance metric between human poses from different camera views. Since direct Euclidean distance is not possible between points in different image planes, they use the point-to-line distance to the epipolar line, forming a symmetrical "epipolar distance" metric.
  3. Energy Minimization: For each pair of cameras, an energy function is defined based on the sum of epipolar distances between corresponding human poses across frames. The temporal displacement (offset) between the two camera views is found by minimizing this energy function.
  4. Multiple Person Association: To handle multiple people, a simple yet effective approach is used: the pose in one view is associated with the pose in another view that has the minimum average epipolar distance to it. This bypasses the need for explicit person re-identification.
  5. Score Regularization: The authors incorporate the confidence scores output by the pose estimator. These scores, typically used for thresholding, are repurposed to regularize the energy function. This "Score" method improves robustness, especially when pose estimations are noisy or incomplete, by weighting more confident detections higher.
  6. Multi-View Synchronization (Graph-Based Global Minimization):
    • Graph Construction: A graph is constructed where each node represents a camera view. The edges between nodes represent the temporal displacement energy between corresponding camera pairs.
  7. Minimum Spanning Tree: A Minimum Spanning Tree (MST) algorithm (Kruskal's) is applied to this graph. The MST finds the optimal global temporal displacement configuration for the entire multi-camera system by minimizing the sum of edge weights (energy values), resolving potential conflicts between pairwise synchronizations.

Datasets

The method is evaluated on four public multi-view datasets:

  • Human3.6M: Features videos from four cameras, with a single subject performing daily actions. It's noted for its high-quality pose data and perfect synchronization (used as a baseline).
  • Shelf: Captures four people interacting in an indoor environment with five cameras. This dataset presents challenges due to occlusion from people and the shelf.
  • Campus: Involves three people in an outdoor square with three cameras. People appear smaller in these images.
  • Store: Uses twelve cameras in a store environment, capturing five people. This dataset has a lower frame rate and partial views.

All datasets provide intrinsic and extrinsic camera parameters. The authors manually offset videos from these datasets to simulate synchronization errors for evaluation.

Key Insights and Ideas

  • Semantic Synchronization: The core idea of using human poses as a semantic signal for synchronization is powerful. It moves beyond relying on external, often costly, hardware or simple visual cues.
  • Epipolar Distance for Pose Alignment: Adapting epipolar geometry to define a distance metric between 2D human poses from different views is a clever way to leverage geometric constraints without requiring 3D information.
  • Confidence Scores as Regularization: Repurposing pose confidence scores to regularize the energy function is a significant improvement. It makes the method more robust to noisy pose estimations, which is common with off-the-shelf detectors and challenging environments.
  • Graph-Based MST for Global Consistency: Using an MST to find a globally optimal synchronization configuration for all cameras is an elegant way to handle interdependencies and avoid local optima.
  • Training-Free Adaptability: The framework's ability to work without any training on the target datasets makes it highly practical for diverse, real-world deployments.

Key Results

The paper demonstrates robust sub-frame synchronization accuracy across all four benchmark datasets.

  • The "Score" method (with confidence score regularization) consistently outperforms the "Simple" method (without score regularization).
  • On the Human3.6M and Campus datasets, the "Simple" method already achieves sub-frame accuracy. The "Score" method further improves this, achieving near-zero error (0.0 and 0.7 respectively).
  • On the Shelf and Store datasets, where pose quality is lower due to occlusion, the "Score" method shows significant improvements, reducing synchronization error from 8.8 to 0.0 and 3.1 to 0.6 respectively. This highlights the robustness gained from score regularization.
  • Ablation studies show that while more frames generally improve accuracy, the "Score" method is more effective at leveraging additional frames, especially in challenging scenarios.
  • Certain actions, like "Walk," where subjects have stable global movements, are more suitable and require fewer frames for synchronization compared to static actions like "Sit."

Limitations and Future Work

  • Pose Estimation Quality: The method's performance is inherently tied to the quality of the 2D human pose estimations. Noisy or incomplete poses can still lead to larger errors, especially in highly occluded scenes.
  • Action Suitability: While the method works for all actions, some actions (e.g., "Sit") require more frames for accurate synchronization due to limited informative motion.
  • Future Work: The authors suggest future work could focus on detecting and classifying different action segments within long videos to assign adaptive weights for better synchronization. They also propose exploring how to detect subjects' actions to assign adaptive weights to clips for improved synchronization.

Summary updated

Abstract

Multiple-camera systems can expand coverage and mitigate occlusion problems. However, temporal synchronization remains a problem for budget cameras and capture devices. We propose an out-of-the-box framework to temporally synchronize multiple cameras using semantic human pose estimation from the videos. Human pose predictions are obtained with an out-of-the-shelf pose estimator for each camera. Our method firstly calibrates each pair of cameras by minimizing an energy function related to epipolar distances. We also propose a simple yet effective multiple-person association algorithm across cameras and a score-regularized energy function for improved performance. Secondly, we integrate the synchronized camera pairs into a graph and derive the optimal temporal displacement configuration for the multiple-camera system. We evaluate our method on four public benchmark datasets and demonstrate robust sub-frame synchronization accuracy on all of them.
BibTeX
@inproceedings{zhang2021semantically,
  author = {Zhe Zhang and Chunyu Wang and Wenhu Qin},
  title = {{Semantically Synchronizing Multiple-Camera Systems with Human Pose Estimation}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=QxJIYluFRm},
}
Figure