Camera networks in plazas, stations and arenas often need to be calibrated without anyone placing a checkerboard. The usual trick is to treat walking people as the calibration object. A pose detector finds their ankles, hips and shoulders in each camera, and the method solves for camera positions that make those body points agree. This works well in sparse scenes and breaks in dense ones. When people block each other, the body points jump around or vanish, and cascaded or perspective-solver calibrations degrade. We propose to use the thing that causes the failure. Whenever one person's silhouette covers another's in some camera, that camera learns which of the two is closer. We call this the occlusion order. It becomes more plentiful as the crowd gets denser, it does not depend on appearance, and it answers the one question a single image cannot: which person is nearer along the line of sight.
The core idea fits in two sentences. Each overlap between two people's masks in a view gives an inequality on their depths from that camera. Thousands of these inequalities across views and time pin down camera placement, break the "tall and far versus short and near" ambiguity for each person, and disambiguate cross-view identity. We add these inequalities as a soft term to a standard spatiotemporal bundle adjustment, which is a joint refinement of cameras, time offsets and 3D body points. We then test whether calibration accuracy stays flat or improves as crowd density rises, where keypoint-only methods get worse.
Core Claim
Monocular scale ambiguity shows up at the level of each person. In one image, a 1.9 m adult at 12 m and a 1.3 m child at 8.2 m can project to nearly the same silhouette. Methods that turn people into metric rulers resolve this with a population height prior. EasyRet3D uses mean human height under a flat ground plane, and CasCalib uses anatomical segment distributions. The research file notes that such priors carry roughly ±15% population variance, and nothing in these pipelines checks the prior against the image. An occlusion order does check it. If the smaller-looking person covers the larger-looking one, the smaller person must be nearer, and so must actually be smaller. Across views, the same pair's order flips depending on which side of them the camera stands, which constrains the camera's center.
The claim is therefore specific. In dense scenes, ordinal depth from mutual occlusion is a measurement that is complementary to 2D keypoints and more reliable than them. Adding it to people-based calibration should reduce rotation, translation and relative-scale error, and the gain should grow with crowd density. Global metric scale still needs one anchor, and we keep the population height prior for that. Occlusion order makes the per-person height estimates consistent with each other, so averaging that prior over many people becomes more trustworthy. We do not claim that occlusion alone recovers meters.
Relation to Prior Work
Four people-based calibration methods supply the machinery we build on, and each fails at the point we target.
- CasCalib splits calibration into a sequence of easier problems: single-view intrinsics from upright poses, a 1D time-offset search on ground-projected ankle tracks, ground-plane rotation plus iterative closest point alignment, then bundle adjustment. It reports failure when 2D noise exceeds about 15 pixels, and it relies on visible ankles.
- Spatiotemporal Multi-Camera Calibration using Freely Moving People jointly estimates cross-view association, rotation and time offset with expectation-maximization over joint directions on the unit sphere. Its authors state that it cannot recover metric scale, and that it conflates people who move in synchrony.
- EasyRet3D lifts each person to 3D with monocular body-mesh models and solves extrinsics with a robust perspective-n-point solver. It downweights occluded views rather than extracting information from them.
- Probabilistic Triangulation treats camera pose as a sampled distribution, but only for a single person.
We reuse the bundle adjustment backbone of the first two methods. What we change is the measurement model: occlusion events become constraints rather than noise.
Two occlusion-aware lines motivate the design but work in different regimes. Crowd4D resolves monocular crowd depth by anchoring feet to a reconstructed terrain surface, and uses a neighbor graph to place occluded pedestrians. It is single-view and needs about 4 hours per 200-frame, 100-person clip. Hi4D and HOLD handle close mutual occlusion with contact and collision terms. Both assume known cameras or a monocular structure-from-motion initialization, and both treat occlusion as an obstacle to reconstruction rather than a source of geometry.
The closest integrated system is CHROMM. It predicts cameras, scene points and body meshes feed-forward and rescales the scene using head–pelvis length ratios, but it has no reasoning about occlusion or contact between people. CaMuViD constrains our framing from the other side. It shows that learned, calibration-free fusion can detect pedestrians under occlusion well, with 95.0% multi-object detection accuracy (MODA) on Wildtrack. Any argument for explicit calibration therefore has to pay off in something CaMuViD cannot produce, such as metric 3D positions, or in transfer to new rigs.
Research Program
The system has three parts.
Measuring occlusion order. For each camera and frame, an off-the-shelf instance segmenter and person tracker give masks. At each boundary pixel shared by two masks, we decide which instance owns the pixel. A second vote comes from the depth ordering of a monocular depth network such as Metric3D v2, using only its ordinal signal and none of its metric values. The two votes are combined into a confidence $w$, and pairs where they disagree are dropped. The output is a set of tuples $(k, t, i, j, u, w)$. Each one says that in camera $k$ at time $t$, person $i$ occludes person $j$ at pixel $u$.
Adding occlusion to the optimization. Let $d_k(\mathbf{X})$ be the depth of 3D point $\mathbf{X}$ along the ray through pixel $u$ from camera $k$. Let $\mathbf{X}_i(t)$ and $\mathbf{X}_j(t)$ be the body-surface points of $i$ and $j$ nearest that ray, taken from the current body estimates. We add
$$L_{\text{occ}} = \sum w \,\mathrm{softplus}\!\left(\frac{d_k(\mathbf{X}_i(t)) - d_k(\mathbf{X}_j(t)) + m}{\tau}\right)$$
to the bundle adjustment objective. Here $m$ is a small margin in scene units and $\tau$ a temperature. We give each person a free size scale, regularized toward a population prior. Without that free scale, the occlusion term would have no per-person ambiguity to resolve.
Using occlusion for initialization and association. At initialization, occlusion orders are scale-free, so they can vote on coarse camera placement around the ground plane before any metric step. We also add an association cue. Two people performing identical motions in sync, which breaks the expectation-maximization method of Lee et al., still have different occlusion relations with their neighbors. The pattern of which neighbors each person covers, and is covered by, then acts as a view-consistent signature.
The artifacts are the modified calibration pipeline, a density-controlled evaluation protocol, and an analysis of when occlusion order is informative. That analysis asks how many independent pairs are needed, and what happens to the half-space constraints on camera centers under near-collinear crowd layouts.
Baselines
The naive reference point is the calibration a practitioner would get with no crowd reasoning. We take the CasCalib cascade with default settings and a fixed 1.7 m mean-height scale anchor, run on the same video.
The strongest existing methods for this question are Lee et al. and CasCalib. They solve exactly our problem, calibration and time synchronization from people without markers, and they report the most directly comparable numbers.
- Lee et al. report a camera rotation error of 0.012 radians and a mean time-offset error under one frame on CMU Panoptic. They remain stable up to 5 pixels of injected joint jitter.
- CasCalib reports an 11% average focal-length error on vPTZ and synchronization within 10 frames on Human3.6M. Its errors rise sharply beyond 15 pixels of noise.
I found no reported calibration result on Lacuna for Wildtrack, MultiviewX, EgoHumans or VirtualCrowd for any of these methods. We will therefore run their released code on those datasets ourselves. All methods get the same detector, segmenter and tracker outputs, the same frames, and the same bundle adjustment solver and iteration budget. Any hyperparameters are tuned on EPFL Terrace, as CasCalib does, with an equal number of tuning trials for every method. Beating Lee et al. on CMU Panoptic means matching 0.012 radians in its sparse regime. Beating it on the dense datasets means lower rotation error at matched density, with the gap widening as density rises.
EasyRet3D serves as the perspective-n-point competitor, using its own occlusion-weighted tracklet stitching. Its reported 25.7 mm Procrustes-aligned mean per-joint position error (PA-MPJPE) on Human3.6M is the reference for downstream 3D pose quality. Human3.6M has little occlusion, so we expect parity there, not gains.
CHROMM is the feed-forward competitor. I found no reported calibration result on Lacuna for CHROMM on any benchmark in the file.
The key ablations share one pipeline and change only the occlusion term:
- the full system with the occlusion term removed;
- the term driven by randomly permuted occlusion labels with the same count, which tests whether gains come from information or from mere regularization;
- the term driven only by Metric3D v2 depth ordering, with no mask evidence.
Why not just…?
Why not just downweight occluded keypoints, as EasyRet3D does? Downweighting discards the frames where the crowd is densest, which are exactly the frames that carry the most occlusion orders. At high density it leaves too few clean keypoints to constrain the cameras. The "occlusion term removed" ablation, with EasyRet3D-style confidence weighting kept on, shows this directly. If that ablation matches the full system across the density sweep, the proposal fails, and we will report that.
Why not just read metric depth from Metric3D v2 and self-calibrate intrinsics with AnyCalib? This is the real threat. AnyCalib recovers intrinsics from one image in about 25 ms. Metric3D v2 reports an absolute relative depth error (AbsRel) of 0.047 on NYUv2 and 0.044 on KITTI, given the correct focal length. Chained together, they give metric foot positions per view, from which extrinsics follow. Our reason for doubt is that those numbers come from indoor rooms and driving scenes. Surveillance crowds at 10–40 m, with pedestrians only tens of pixels tall, are different, and small relative depth errors between adjacent people are exactly what ordering needs to get right. We run this chain as a full baseline. If it matches our rotation and scale errors on Wildtrack and MultiviewX, the occlusion term adds nothing beyond a strong depth prior, and the contribution shrinks to the mask-evidence gating.
Why not just skip calibration and use CaMuViD? For 2D detection that may well be the right answer, given its 95.0% MODA on Wildtrack and 86.4% in synthetic-to-real transfer. But CaMuViD produces no camera poses, metric positions or 3D bodies. The comparison that matters is downstream. We feed our recovered calibration to a standard geometric bird's-eye-view detector and ask whether it closes the gap to CaMuViD, and whether its synthetic-to-real transfer beats the 76.7% geometry-dependent baseline CaMuViD cites.
Why not just average the height prior over more people? Averaging reduces variance only if per-person size errors are independent and unbiased. Depth confusion correlates them, because a whole cluster can be pushed collectively nearer or farther. We test this on synthetic data by measuring per-person height error with and without the occlusion term. If plain averaging gives the same global scale error, we will say so.
Proposed Experiments
The main experiment is a density sweep. On MultiviewX and VirtualCrowd, which provide ground-truth calibration, 3D positions and body sizes, we bin frame windows by the number of occluding pairs per view. Within each bin we report rotation error, translation direction error, global scale error and per-person height error. On Wildtrack, which provides ground-truth calibration, we report the same calibration errors without height error. The decisive pattern is error curves over density: keypoint-only baselines rising, ours flat or falling. A flat gap would mean occlusion order is merely redundant.
Secondary checks cover other regimes:
- CMU Panoptic. A sparse regime where we must not regress from Lee et al.'s 0.012 radians.
- Hi4D, with its studio calibration held out. Two people in close contact, the extreme of mutual occlusion, testing whether occlusion order still helps when only two people are present.
- Synchronized motion. A synthetic case built by duplicating trajectories on MultiviewX, to test the association claim against Lee et al.'s known failure mode.
- EgoHumans. Real in-the-wild validation of scale and pose against its visual-inertial trajectories.
Across all settings, each method is judged on three things: calibration error, downstream 3D pose error, and bird's-eye-view detection MODA using the recovered cameras.
References
- EasyRet3D: Uncalibrated Multi-View Multi-Human 3D Reconstruction and Tracking
- CasCalib: Cascaded Calibration for Motion Capture from Sparse Unsynchronized Cameras
- Spatiotemporal Multi-Camera Calibration Using Freely Moving People
- Probabilistic Triangulation for Uncalibrated Multi-View 3D Human Pose Estimation
- Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction
- Hi4D: 4D Instance Segmentation of Close Human Interaction
- HOLD: Category-Agnostic 3D Reconstruction of Interacting Hands and Objects from Video
- Coherent Human-Scene Reconstruction from Multi-Person Multi-View Video in a Single Pass
- CaMuViD: Calibration-Free Multi-View Detection
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- AnyCalib: On-Manifold Learning for Model-Agnostic Single-View Camera Calibration