Multiple-Camera Tracking of Rigid Objects

Frédérick Martin, Radu HoraudarXiv 2026 · DOI · arXiv

This paper, published in the International Journal of Robotics Research in 2002, presents a robust framework for tracking complex 3D rigid objects using one or more cameras. By treating tracking as a "visual servoing" problem, the authors demonstrate how to align a 3D model with image contours while simultaneously updating camera calibration parameters.

TL;DR

The authors propose a visual servoing approach to track rigid 3D objects by minimizing the error between predicted model edges and detected image contours. The system is robust to occlusions and camera singularities by leveraging multiple cameras in parallel without requiring traditional stereo correspondences or overlapping fields of view.

What Problem Does This Paper Address?

Tracking a 3D object from 2D images is inherently difficult due to ambiguities and "singularities"—specific geometric configurations where a camera cannot distinguish between different types of motion (e.g., moving away vs. zooming out).

While multi-camera systems can resolve these issues, most traditional approaches treat them as stereo devices. This requires "image-to-image" correspondences, meaning the same feature must be visible and matched across at least two cameras. This is hard to maintain in complex environments or when cameras are placed far apart. Furthermore, many trackers assume cameras are perfectly calibrated off-line, failing when internal parameters like focal length (zoom) change during operation.

Methods

The core of the approach is a visual servoing loop. Instead of just finding the object's pose, the system treats the 3D model as a virtual robot. At each frame, it calculates a "kinematic screw" (velocity vector) that moves the model to minimize the distance between its projected edges and the actual edges found in the image.

Key methodological components include:

  • Point-to-Contour Matching: Rather than trying to find exact point matches, the algorithm projects model edges and searches for image contours along the normal direction to those edges. This is much more robust than point-to-point matching.
  • Jacobian Analysis: The authors provide a rigorous algebraic analysis of the Jacobian matrix—the mathematical engine that relates model motion to image error. They identify the minimum number of matches needed for stability and define the "singular" configurations that lead to tracking failure.
  • Parallel Multi-Camera Tracking: Instead of stereo matching, the system runs several single-camera trackers in parallel. The only constraint is the object's rigidity; as long as the relative positions of the cameras are known, information from different viewpoints is combined into a single global pose estimate.
  • On-line Calibration: The state vector includes internal camera parameters (focal length $f$, and optical center $u_0, v_0$). The tracker updates these parameters at every iteration, allowing it to handle zooming and focus changes on the fly.
Figure 1: The strategy for point-to-contour matching. A normal is raised from the predicted model point $m^0$ to find the intersection with the image contour at $m$. The distance $d$ is then minimized.

Datasets

The authors evaluated their method using a complex industrial "ship part" characterized by approximately 300 straight and curvilinear edges.

  • Hardware: The setup used up to six black-and-white cameras connected to a PC (Pentium 3, 733MHz).
  • Benchmarks: No standard public datasets were used (reflecting the era of publication); instead, the authors performed extensive experiments involving manual initialization, varying focal lengths, and deliberate occlusions.

Key Insights and Ideas

  • Mathematical Linkage: The authors proved that point-to-contour tracking shares the same mathematical singularities as point-to-point tracking. This theoretical grounding allows researchers to predict exactly when a contour-based tracker will fail.
  • Singularity Resolution via Multi-View: A major "aha moment" is the demonstration that single-camera singularities (like the inability to distinguish depth from zoom when viewing a planar object head-on) disappear when a second camera views the object from a different angle.
  • Distributed Information: Because the system doesn't require image-to-image matches, cameras can be placed arbitrarily. One camera might see only the top of an object while another sees only the side; the rigid-body constraint allows the system to fuse this "distributed" information effectively.
  • Tracking as Calibration: The paper shows that a robust tracker is essentially a continuous calibration procedure. By minimizing image error, the system naturally converges toward the true internal camera parameters.

Key Results

  • Real-time Performance: The system achieved 25Hz with one camera, 12Hz with three cameras, and 6Hz with six cameras on 2002-era hardware.
  • High Accuracy: For static objects, the pose converged to an average image error of just 0.8 pixels within 18 iterations.
  • Robustness to Occlusion: The multi-camera setup successfully maintained tracking even when one camera was totally occluded and others suffered partial occlusions.
  • Zoom Distinction: Experiments confirmed the tracker could accurately distinguish between an object moving away and a camera zooming out, a classic ambiguity in monocular vision.

Limitations and Future Work

  • Edge Dependency: The method relies entirely on visible edges. In environments with low contrast or "texture-less" objects, the tracker may struggle to find valid correspondences.
  • Initialization: The initial pose of the model must be relatively close to the actual object position (set manually in these experiments) for the local minimization to succeed.
  • Future Directions: The authors suggest incorporating texture information (image intensity patterns) alongside edge information to improve robustness in complex natural or industrial scenes.

Summary updated

Abstract

In this paper we describe a method for tracking rigid objects using one or several cameras. The tracking process consists of aligning a 3-D model representation of an object with image contours by measuring and minimizing the image error between predicted model points and image contours. The tracker behaves like a visual servo loop where the internal and external camera parameters are updated at each new image acquisition. We study in detail the Jacobian matrix associated with this minimization process in the presence of both point-to-point and point-to-contour matches. We establish the minimal number of matches that are needed as well as the singular configurations leading to a rank-defficient Jacobian matrix. We find a mathematical link between the point-to-point and point-to-contour cases. Based on this link we show that the latter has the same kind of singularities than the former. Moreover, we study multiple camera configurations which optimize the robustness of the method in the presence of single-camera singularities, bad, noisy, or missing data. Extensive experiments done with a complex ship part and with up to three cameras validate the method. In particular we show that the tracker may well be used as a camera calibration procedure.
BibTeX
@misc{martin2026multiple,
  author = {Frédérick Martin and Radu Horaud},
  title = {{Multiple-Camera Tracking of Rigid Objects}},
  year = {2026},
  eprint = {2611.25904},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25904},
}
Figure