When a single photo shows several people, every reconstructed body has one degree of freedom the image cannot fix. A person can be small and near or large and far, and both produce the same picture. Current systems handle this in one of three ways. They assume everyone is average height, they snap everyone's feet to one fitted floor plane, or they read each person's distance off a depth-estimation network. Each fails in a familiar setting: a child next to an adult, people sitting or jumping, or two people hugging, where a few centimetres of depth error makes their bodies pass through each other.
This note proposes using the people as rulers for one another. Three kinds of relation between pairs of people carry metric information. Occlusion says who is in front. Contact says two surfaces touch. Non-penetration says two bodies cannot share volume. Uniformly scaling the whole scene preserves all three relations, so they cannot recover absolute scale. They do strongly constrain each person's scale relative to everyone else. We separate the problem along that line. Pairwise relations, predicted by a network that fuses image features with depth-model features, fix the relative layout. A single global scale factor, pooled across all people from a depth model and from height statistics, fixes the absolute size.
Core Claim
Represent each detected person $i$ by a log-depth $t_i = \log Z_i$. The person's body scale is tied to $t_i$ so that their 2D projection stays fixed, which means the depth-size ambiguity is a one-dimensional slide along the camera ray. Pairwise physical relations are then constraints on differences $t_i - t_j$ and are invariant to a shared shift of all $t_i$. Everything a single image leaves undetermined about the group reduces to one global scalar, plus whatever per-person freedom the relations fail to pin down.
Our claim is that, in multi-person scenes with interaction, solving for the $t_i$ under predicted three-way relations gives more accurate relative depth and relative height than two alternatives. The first is reading metric depth per person. The second is ground-plane snapping. Both comparisons are strongest on frames where feet are off the floor or out of view. The three-way relations are "i in front," "j in front," and "touching." The solver also enforces non-penetration and pools a global scale across people.
The third relation, "touching," directly addresses a known weakness of ordinal supervision. A ranking loss gives zero gradient when two surfaces are at the same depth, which is exactly the contact case.
Research Program
The system has three parts.
Initial bodies. An off-the-shelf multi-person mesh regressor provides poses and shapes. Each person gets an explicit scale scalar separate from the body-shape parameters, following the decoupling in Body Size and Depth Disambiguation in Multi-Person Reconstruction from Single Images.
Relation predictor. This is the cross-modal fusion component. For each pair of people it fuses three inputs: appearance features of the two crops and their union, intermediate features from a frozen depth model, and the rendered overlap of the two initial meshes. The design follows the confidence-gated depth-appearance fusion of Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery, but applied to pairs of people rather than one tracked person. For each pair it outputs a distribution over the three relations. For each person it also outputs an uncertainty $\sigma_i$ on their depth-model reading.
Training labels come from two sources. On multi-person 3D training data such as MuCo-3DHP and Panoptic, relations are computed from ground-truth meshes. On in-the-wild images, weak ordinal labels come from instance-mask overlaps, as in Coherent Reconstruction of Multiple Humans From a Single Image.
We fuse depth features rather than using raw depth values for a specific reason. Depth maps blur at person boundaries, which is exactly where front/behind/touching is decided. The appearance stream sees occlusion cues such as a hand over a shoulder or a clothing fold that the depth map smooths away.
Small differentiable solver. The solver minimizes
$$E(t)=\sum_i \frac{(t_i-\log D_i)^2}{\sigma_i^2} + \frac{(\bar h(t)-\mu_h)^2}{\sigma_h^2/N} + \sum_{i<j}\Big[p^{o}_{ij}\,\mathrm{softplus}(t_i-t_j+m) + p^{c}_{ij}\,g_{ij}(t)^2 + \lambda\,P_{ij}(t)\Big]$$
The symbols are defined as follows. $D_i$ is the metric depth the frozen depth model reports at person $i$. $\bar h(t)$ is the group's mean log-height implied by the scales, and $\mu_h, \sigma_h$ are the population mean and spread of log-height. $N$ is the number of people; dividing the variance by $N$ reflects that averaging $N$ roughly independent heights shrinks variance by $1/N$. $p^{o}_{ij}$ and $p^{c}_{ij}$ are the predicted probabilities that $i$ is in front of $j$ and that they touch, and $m$ is a small ordering margin. $g_{ij}$ is the metric gap between the two closest surface points. $P_{ij}$ is a robust interpenetration penalty computed from signed distance fields.
The first two terms are the only ones that move the global scale. All pairwise terms are shift-invariant. This removes a failure described in the literature: in an unscaled frame, a collision penalty can be "satisfied" by dilating the whole scene. Here it cannot. After the $t_i$ settle, a short pose refinement under the same contact and non-penetration terms fixes the remaining limb-level clipping. Visibility weighting keeps the penetration term away from occluded vertices, the failure mode reported by PhySIC.
The work stays deliberately narrow: single images, people only, and rigid human-human relations. The analysis question is how much relative-scale information the pairwise relations carry beyond what a depth model already gives. We measure this per relation type and as a function of how far apart people are.
Baselines
Average-height reference. The naive point is the initial regressor with every person given average height and placed at the depth that height implies. This is what most single-person pipelines implicitly do. It is the floor every other method must clear, and it shows the depth-size trap directly on child/adult pairs.
Ground-plane snapping. The strongest existing method on the exact question is Body Size and Depth Disambiguation in Multi-Person Reconstruction from Single Images. It fits a floor from a monocular depth map with RANSAC and optimizes per-person scale so that feet touch the floor. It reports 95.58% depth-ordering accuracy on MuPoTS-3D, up from 92.20% for the prior best, and 57.75% relative-height accuracy ($h_{\text{ord}}$). Its authors also found that once feet are grounded, collision losses become largely unnecessary, which is a direct challenge to our use of non-penetration.
We will run it with the same initial bodies and the same depth model, so that differences come from the scale-resolution step alone. Beating it means two things. On full MuPoTS-3D, we must at least match its 95.58% ordering and exceed its 57.75% $h_{\text{ord}}$. On a feet-not-grounded subset, we must clearly exceed it on both metrics, because that subset is where its assumption breaks.
Per-person metric depth readout. This is the PhySIC-style approach: place each person at the metric depth from a zero-shot metric depth model, in the manner of PhySIC or Metric3D v2 (Zhang et al.). It is the most likely real threat. PhySIC reports results only on single-person human-scene data (PROX, PA-MPJPE 42 mm). We found no reported multi-person relative-depth result on Lacuna for this readout, so we will compute it ourselves on MuPoTS-3D and Panoptic with the identical depth backbone.
Learned regressors with ordinal and collision losses. These are Coherent Reconstruction of Multiple Humans From a Single Image, which reports 3DPCK 69.12 against 66.95 and cuts interpenetrating instances from 114 to 34 on MuPoTS-3D, and Body Meshes as Points, which reports Panoptic MPJPE 135.4 mm against 143.2 mm. Our solver will be applied on top of each, under matched training data.
Ablations. Four ablations isolate the key ingredients:
- Drop the "touching" class, leaving a pure two-way ordinal model.
- Drop the depth-model term, so global scale comes from pooled height alone.
- Replace the fused relation predictor with relations obtained by thresholding depth-map differences. This tests whether fusion is doing anything.
- Supply oracle relations from ground truth. This gives the ceiling of the solver.
Why not just…?
Why not just read each person's depth from a metric depth model? This is the serious competitor, and it may win when people stand far apart. Our argument is about error structure, not average error. Depth-model readings on a person are a blur over the silhouette and its boundary, and independent per-person errors of a few percent at 3–5 m already exceed the gap between two people in contact. Pairwise relations constrain differences directly.
The decisive experiment bins MuPoTS-3D and Panoptic person pairs by ground-truth separation. In each bin we compare the error in $t_i - t_j$ for the depth readout against our solver. If the readout matches us even in the closest-separation bins, the core claim is wrong and we will report it as such.
Why not just snap feet to a fitted floor? It works well when its assumption holds, and its 95.58% ordering result shows that. But it needs one flat floor and visible standing feet. Its own limitations name stairs, slopes, sitting, and jumping as failure cases. Our feet-not-grounded subset of MuPoTS-3D and 3DPW is built to test exactly this. We define it as frames where some ground-truth ankle lies well above the lowest ground-truth foot, or where ankles are cropped. We will also run the two methods combined, with the floor term added to $E$, because if they are complementary that combination is the practical answer.
Why not just add a collision loss to an end-to-end regressor, as Coherent Reconstruction does? Its penetration loss operates in an unanchored frame, and its authors state that the model still cannot separate a small near person from a large far one. A feed-forward regressor also cannot propagate one confident relation through a chain of people. For example, if A touches B and B occludes C, that constrains A relative to C, and a joint solve captures this. Ablation 1 and the comparison on top of Coherent Reconstruction measure whether the explicit solve adds anything beyond the trained-in losses.
Proposed Experiments
Datasets and metrics. - MuPoTS-3D is the main testbed. We report depth-ordering accuracy, $h_{\text{ord}}$, and pairwise normalized distance $d_{\text{norm}}$ as defined in Body Size and Depth Disambiguation, together with 3DPCK, interpenetrating-instance count, and absolute root-position error. - CMU Panoptic adds dense social scenes, scored by MPJPE and the same relative metrics. - 3DPW supplies sitting and climbing frames for the feet-not-grounded subset. - We will also report contact precision and recall, computed from ground-truth meshes, on interacting pairs in MuPoTS-3D and Panoptic.
The file lists no dedicated close-interaction benchmark. Our contact evaluation is therefore limited to interactions those datasets happen to contain, and we treat it as secondary evidence.
Decisive checks. 1. Separation-binned relative-depth error. Our solver must beat the metric-depth readout in the close-separation bins. Otherwise the idea fails. 2. Grounded versus feet-not-grounded split. We must clearly exceed ground-plane snapping on the not-grounded subset without losing more than noise on the grounded one. 3. Oracle-relation ceiling against predicted relations. The gap between them tells us whether the bottleneck is the relation predictor or the solver. 4. Shift-invariance test. We rescale the depth-model input by a constant and confirm that relative metrics stay unchanged while absolute error moves. This verifies empirically that pairwise terms carry no global scale, as the formulation implies.
A positive result on checks 1 and 2 would establish interpersonal physical relations as a scale signal that needs neither a floor nor a calibrated sensor.
References
- Body Size and Depth Disambiguation in Multi-Person Reconstruction from Single Images
- Depth-Guided Metric-Aware Temporal Consistency for Monocular Video Human Mesh Recovery
- Coherent Reconstruction of Multiple Humans From a Single Image
- PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Body Meshes as Points