Estimate 3D Camera Pose from 2D Pedestrian Trajectories

Yan Xu, Vivek Roy, Kris KitaniWACV 2020

TL;DR

This paper proposes a novel data-driven approach to estimate the 3D pose of a static camera using only 2D pedestrian trajectories. By training a neural network on synthetic data, the method can directly infer camera pose from real-world pedestrian movements, offering a cost-effective and accurate alternative to traditional methods, especially when 3D point annotations are unavailable.

What Problem Does This Paper Address?

Static surveillance cameras, while seemingly fixed, can experience shifts in their 3D pose due to external factors like wind or vibrations. Re-estimating this pose is crucial for accurate scene understanding and tracking. Traditional methods like Perspective-n-Point (PnP) require 2D-to-3D feature correspondences, which are expensive and time-consuming to obtain, especially for continuous pose estimation. While geometric methods can infer pose from constraints like parallelism, they often struggle when such features are absent. This paper addresses the challenge of camera pose estimation when 3D point information is unavailable, proposing to leverage the inherent geometric information within 2D pedestrian trajectories.

Methods

The core of the proposed method is a neural network (NN) regressor trained to learn a direct mapping from 2D pedestrian trajectories to the 3D camera pose. The NN architecture consists of:

  • RNN Feature Encoder (FE): A bi-directional LSTM processes the input 2D trajectory (a sequence of (x, y) coordinates) to extract a fixed-dimensional feature representation.
  • Joint-feature Extractor (JE): A multi-layer perceptron (MLP) takes the FE output and learns a common feature that aids in predicting both location and orientation.
  • Location Branch (LB) & Orientation Branch (OB): Separate MLPs predict the camera's 3D location (specifically, its height, as other dimensions are ambiguous from 2D trajectories alone) and orientation (as a quaternion).

A key aspect is the use of synthetic data for training. The authors generate synthetic pedestrian trajectories by simulating camera poses around an initial estimate and using a human motion model. This bypasses the need for costly real-world data annotation. The loss function combines a Euclidean distance for location error and a quaternion-based error for orientation.

Datasets

The paper uses two primary datasets for evaluation:

  • Town Centre Street (TCS) [3]: An outdoor dataset with video sequences of a busy street, providing camera intrinsics, extrinsics, and bounding boxes.
  • DUKEMTMC (DM) [39]: A large, manually annotated, and calibrated multi-camera dataset from Duke University, offering ground plane coordinates for pedestrians.

The authors generated synthetic training data using these datasets as a basis, simulating camera poses and pedestrian trajectories.

Key Insights and Ideas

  • Pedestrian Trajectories as Rich Cues: The paper demonstrates that 2D pedestrian trajectories, often overlooked, contain sufficient geometric information to infer the 3D pose of a static camera.
  • Synthetic Data for Practicality: Training a robust NN regressor solely on synthetic data and then applying it directly to real-world data is feasible and significantly reduces annotation costs. This highlights the generalizability of the learned features.
  • Ambiguity Handling: The method acknowledges the inherent ambiguity in inferring camera height and yaw from 2D trajectories and explicitly addresses this by predicting only the camera height and fixing the yaw angle.
  • Separate Prediction Branches: Using separate branches for location and orientation prediction in the NN architecture leads to higher accuracy than a single combined output.

Key Results

  • Accuracy Improvement: The proposed method achieves approximately 50% improvement in both position and orientation prediction accuracy compared to state-of-the-art (SOTA) learning-based baselines (PoseNet [20]) and geometric methods (VP calibration [17]).
  • Synthetic-to-Real Transfer: The NN trained on synthetic data performs exceptionally well on real data, validating the effectiveness of the synthetic data generation strategy.
  • Efficiency: The training time cost of the proposed method is exponentially lower than fine-tuned baseline models, even outperforming baselines without fine-tuning.
  • Robustness: The method shows stable and accurate results across various scenes and conditions, outperforming DSAC++ [5] which struggles with the sparse input data.

Limitations and Future Work

  • Assumption of Static Camera: The current method assumes a static camera, and its performance might degrade if the camera undergoes significant motion during the observation period.
  • Pedestrian Speed Assumption: While the method leverages the statistical average of pedestrian speeds, variations in speed can impact accuracy, particularly for location prediction. The paper notes that trajectory height variations also negatively affect location accuracy.
  • Future Work: The authors suggest exploring more sophisticated human motion models and investigating methods to handle dynamic camera scenarios. Further research into improving robustness to varying pedestrian speeds and trajectory heights could also be beneficial.

Summary updated

BibTeX
@inproceedings{xu2020estimate,
  author = {Yan Xu and Vivek Roy and Kris Kitani},
  title = {{Estimate 3D Camera Pose from 2D Pedestrian Trajectories}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure