DeepPTZ: Deep Self-Calibration for PTZ Cameras

Chaoning Zhang, François Rameau, Junsik Kim, Dawit Mureja Argaw, Jean-Charles Bazin, In So Kweon

DeepPTZ is a pioneering deep learning framework designed to self-calibrate Pan-Tilt-Zoom (PTZ) cameras using only a pair of images. By employing a novel "Dual-Siamese" network architecture, it simultaneously estimates focal lengths, lens distortion, and relative rotation angles, outperforming both traditional geometric methods and previous single-image deep learning approaches.

The Challenge of PTZ Calibration

PTZ cameras are staples of modern surveillance and robotics because they can rotate (pan/tilt) and zoom in to capture fine details. However, this flexibility is a double-edged sword: every time a camera zooms, its internal geometry—specifically its focal length and lens distortion—changes.

To use these images for tasks like 3D reconstruction or seamless image stitching, the camera must be "calibrated." Traditional calibration involves showing the camera a known pattern (like a checkerboard), which isn't feasible for a camera mounted on a high pole in a city center. "Self-calibration" attempts to solve this using only the images captured, but traditional methods often fail if the images have heavy distortion or if the zoom level changes significantly between shots. DeepPTZ fills this gap using a robust, data-driven deep learning approach.

Methodology: The Dual-Siamese Network (DSNet)

The researchers propose a specialized CNN architecture called the Dual-Siamese Network (DSNet). It takes two successive images ($I_1, I_2$) as input and outputs seven parameters: the focal lengths for both images ($f_1, f_2$), the radial distortion parameters ($\xi_1, \xi_2$), and the relative rotation (yaw, pitch, roll) between the two views.

Architecture of the Single Siamese Network component showing feature extraction and combining layers.

The architecture is built upon several key innovations:

  1. Siamese Backbone: The network uses two weight-sharing Inception-v3 branches to extract features from both images.
  2. Correlation Layer: Instead of just stacking features, the authors use a correlation (combining) layer. This explicitly forces the network to look for matching patterns between the two images, mimicking the "feature matching" step used in traditional computer vision.
  3. Bidirectional Constraint: This is the "Dual" part of DSNet. The network processes the pair ($I_1, I_2$) to get a forward estimate, then flips the order to ($I_2, I_1$) for a backward estimate. By enforcing consistency between these passes, the model achieves much higher stability and accuracy.

Key Results and Performance

Because no massive dataset of calibrated PTZ image pairs existed, the authors built their own using the SUN360 database, generating 128,000 image pairs with varying zoom, rotation, and lens distortion.

Cumulative error distributions for rotation ($y_{12}$) and focal length ($f_1$) comparing DSNet to traditional geometric methods.

DSNet demonstrated significant improvements over existing methods:

  • Accuracy: On a test set with 299x299 pixel images, DSNet achieved a mean focal length error of roughly 11.4 pixels—significantly better than the ~41 pixels seen in single-image methods.
  • Robustness: When compared to traditional geometric methods (like the 3-point method), DeepPTZ was much more resilient to lens distortion. As shown in the cumulative error plots, DSNet maintains high accuracy where traditional methods often break down.
  • Implicit Ensembling: By averaging the forward and backward predictions from the Dual-Siamese structure during testing, the model achieves an "ensemble effect," reducing error by more than 10% without needing to train multiple separate models.

Conclusion and Future Directions

DeepPTZ proves that multi-view deep learning provides vital geometric constraints that drastically reduce calibration error. While the current model assumes pure rotation, future work could involve updating the geometric model to include translation (epipolar) constraints or using sequence-based models like Transformers to maintain calibration over time.

Summary updated

Abstract

Rotating and zooming cameras, also called PTZ (Pan-Tilt-Zoom) cameras, are widely used in modern surveillance systems. While their zooming ability allows acquiring detailed images of the scene, it also makes their calibration more challenging since any zooming action results in a modification of their intrinsic parameters. Therefore, such camera calibration has to be computed online; this process is called self-calibration. In this paper, given an image pair captured by a PTZ camera, we propose a deep learning based approach to automatically estimate the focal length and distortion parameters of both images as well as the rotation angles between them. The proposed approach relies on a dual-Siamese structure, imposing bidirectional constraints. The proposed network is trained on a large-scale dataset automatically generated from a set of panoramas. Empirically, we demonstrate that our proposed approach achieves competitive performance with respect to both deep learning based and traditional state-of-the art methods. Our code and model will be publicly available at https://github.com/ChaoningZhang/DeepPTZ.
BibTeX
@inproceedings{zhang2020deepptz,
  author = {Chaoning Zhang and François Rameau and Junsik Kim and Dawit Mureja Argaw and Jean-Charles Bazin and In So Kweon},
  title = {{DeepPTZ: Deep Self-Calibration for PTZ Cameras}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093629},
}
Figure