Automatic Multi-Camera Extrinsic Parameter Calibration Based on Pedestrian Torsors †

Anh Minh Truong, Wilfried Philips, Nikos Deligiannis, Lusine Abrahamyan, Junzhi Guan

TL;DR

This paper introduces a novel, fully automatic method for calibrating multiple cameras in a network by using pedestrians as natural calibration objects. By analyzing human pose estimation data, the system can accurately determine camera extrinsics, even in challenging scenarios with occlusions or multiple people, significantly reducing the need for manual calibration objects and effort.

What Problem Does This Paper Address?

Accurately knowing the relative positions and orientations (extrinsic parameters) of cameras in a network is crucial for many computer vision tasks like tracking, 3D reconstruction, and augmented reality. Traditional methods rely on placing specific calibration objects within the scene, which is often impractical, costly, and time-consuming, especially when cameras are moved or recalibration is needed. This is particularly problematic for field deployments or when dealing with existing video footage where calibration objects were not present. The paper aims to solve this by proposing a method that leverages naturally occurring elements in the scene – pedestrians – as the calibration targets.

Methods

The core of the proposed method is to use human pose estimation to identify key points of pedestrians (specifically, the top of the head/neck and the bottom of the feet or hips) in multiple camera views.

  • Pedestrian as Calibration Object: Instead of artificial targets, pedestrians are treated as "walking sticks" or vertical elements. Their detected 2D positions in images are used to infer their 3D positions.
  • Human Pose Estimation: The paper utilizes modern human pose estimation techniques (like OpenPose) to robustly detect the skeleton of pedestrians, providing accurate locations for joints like the neck (top) and ankles/hips (bottom).
  • Correspondence Matching: For multi-camera calibration, a brute-force but efficient method is employed to find correspondences between pedestrians observed in different cameras. This involves generating all possible pairs and selecting the most consistent ones based on estimated extrinsic parameters.
  • 3D Pose Estimation: Using the 2D correspondences and estimated camera intrinsics, the 3D positions of the pedestrian's top and bottom are computed.
  • Extrinsic Parameter Computation: The 3D pedestrian poses are then used to estimate the rigid body transformation (rotation and translation) between camera coordinate systems.
  • Handling Occlusions: A novel approach is introduced to calibrate using only the top of the pedestrian and their centerline when the bottom is heavily occluded. This is achieved by estimating a plane defined by the pedestrian's top points and using homography decomposition.
  • Robustness: The method incorporates a random sampling strategy to mitigate the effects of noisy pose estimations and outliers, improving overall calibration accuracy and reducing the amount of data needed.
  • Joint Refinement: Finally, the pairwise calibrated parameters are jointly refined to minimize the total reprojection error across all cameras in the network.

Datasets

The paper evaluates its method on several datasets:

  • Camera Network 1 (CN1): A custom setup with four side-view cameras in a controlled environment (empty room).
  • Camera Network 2 (CN2): A custom setup with three cameras in a kitchen, used to test robustness to occlusions during household tasks.
  • EPFL-Terrace dataset (CN3): A public multi-camera pedestrian dataset with four DV cameras, used for evaluating performance with multiple people.
  • Camera Network 4 (CN4): A custom network with five cameras (varying heights) in an empty room, used for comparing single vs. multiple pedestrian calibration.
  • Camera Network 5 (CN5): A real-world intersection scene in Ghent, used to test calibration in uncontrolled environments with small, distant pedestrians.
  • PETS2009-S2L1 dataset (CN6): A public dataset for crowded scenes with multiple pedestrians, used to evaluate performance with low-resolution pedestrians.
  • Camera Network 7: A heavily occluded office scene, used to test the method's ability to calibrate without feet/hip information.

Key Insights and Ideas

  • Pedestrians as Ubiquitous Calibration Targets: The most significant insight is the shift from artificial calibration objects to naturally occurring pedestrians. This dramatically simplifies deployment and recalibration.
  • Leveraging Human Pose Estimation: The paper effectively demonstrates how advancements in human pose estimation can be directly applied to solve a fundamental computer vision problem like camera calibration.
  • Robustness to Occlusion: The development of a method that can calibrate using only the top and centerline of a pedestrian is a key innovation, making it applicable in many real-world scenarios where lower body parts are obscured.
  • Automatic and Efficient Calibration: The proposed method is fully automatic, eliminating manual annotation and significantly reducing calibration time, especially compared to traditional methods.
  • Random Sampling for Outlier Rejection: The use of random sampling to select key locations improves robustness and reduces the data collection time required for accurate calibration.

Key Results

The proposed method achieves high accuracy, with triangulation errors in the order of a few centimeters in controlled environments.

  • Accuracy: Triangulation errors as low as 1.33 cm (CN1, feet) and 2.1 cm (CN3, feet/hip) were reported. Relative reprojection errors were also consistently low, often below 5%.
  • Speed: Calibration can be achieved in as little as one minute in controlled environments and a few minutes in uncontrolled settings. For complex datasets like EPFL-Terrace, it took approximately 210-270 seconds.
  • Robustness: The method performs well across various scenarios, including single and multiple pedestrians, occlusions, and real-world intersections. The method using only top and centerline information also yields comparable accuracy to methods using hip joints when feet are occluded.
  • Comparison: The proposed method generally outperforms or matches state-of-the-art methods like Guan et al. [14] and Hödlmoser et al. [24] in terms of accuracy and robustness.

Limitations and Future Work

  • Pedestrian Size and Pose Stability: The paper notes that in intersection scenarios, pedestrians can appear very small, making pose estimation less stable. Similarly, if a pedestrian's posture changes significantly during observation, it can affect calibration accuracy.
  • Short Trajectories: The authors mention that if walking trajectories are too short, it can lead to insufficient data and potentially higher relative reprojection errors.
  • Future Work: The authors plan to investigate a "regional selection method" to address the issue of short walking trajectories, aiming to automatically identify the most informative parts of the video for calibration.

Summary updated

Abstract

Extrinsic camera calibration is essential for any computer vision task in a camera network. Typically, researchers place a calibration object in the scene to calibrate all the cameras in a camera network. However, when installing cameras in the field, this approach can be costly and impractical, especially when recalibration is needed. This paper proposes a novel, accurate and fully automatic extrinsic calibration framework for camera networks with partially overlapping views. The proposed method considers the pedestrians in the observed scene as the calibration objects and analyzes the pedestrian tracks to obtain extrinsic parameters. Compared to the state of the art, the new method is fully automatic and robust in various environments. Our method detect human poses in the camera images and then models walking persons as vertical sticks. We apply a brute-force method to determines the correspondence between persons in multiple camera images. This information along with 3D estimated locations of the top and the bottom of the pedestrians are then used to compute the extrinsic calibration matrices. We also propose a novel method to calibrate the camera network by only using the top and centerline of the person when the bottom of the person is not available in heavily occluded scenes. We verified the robustness of the method in different camera setups and for both single and multiple walking people. The results show that the triangulation error of a few centimeters can be obtained. Typically, it requires less than one minute of observing the walking people to reach this accuracy in controlled environments. It also just takes a few minutes to collect enough data for the calibration in uncontrolled environments. Our proposed method can perform well in various situations such as multi-person, occlusions, or even at real intersections on the street.
BibTeX
@inproceedings{truong2019automatic,
  author = {Anh Minh Truong and Wilfried Philips and Nikos Deligiannis and Lusine Abrahamyan and Junzhi Guan},
  title = {{Automatic Multi-Camera Extrinsic Parameter Calibration Based on Pedestrian Torsors †}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=KxeGeobgJ39c},
}
Figure