Body of Evidence: Triangulating 3D Human Pose with Geometry-Conditioned Depth

Erik Brorsson, Kilian Freitag, Josef Bengtson, Kristofer Bengtsson, Lars Hammarstrand, Knut Åkesson, Fredrik KahlarXiv 2026 · arXiv

TL;DR

Body of Evidence (BoE) is a novel 3D human pose estimator that solves a classic problem in multi-camera systems: what happens when a joint is only visible from a single camera? By predicting per-camera depths and integrating them into a new Learnable Depth-Aware Triangulation (LDAT) framework, BoE can reconstruct 3D poses even under severe occlusion or sparse camera coverage, outperforming previous methods by over 10 points in challenging three-camera setups.

What Problem Does This Paper Address?

In traditional multi-view 3D pose estimation, researchers rely on triangulation. If two or more cameras see the same joint, they can find its 3D position by looking at where the viewing rays intersect. However, in the real world—think crowded sports arenas or sparse industrial monitoring—joints are frequently occluded or only visible to one camera.

When a joint is seen by only one camera, we know the ray it lies on, but its depth along that ray is a mystery. This makes the 3D reconstruction "underdetermined." While end-to-end deep learning methods try to guess the 3D pose directly, they often fail to generalize when you move the cameras around. BoE aims to keep the mathematical strengths of triangulation while giving it the "evidence" it needs to handle these single-view cases.

Methods

The authors introduce an iterative framework that treats 3D pose estimation as a conversation between image features and 3D geometry.

  1. Iterative Refinement: The model starts with an initial 3D pose guess and projects it into all camera views to extract image evidence. This process repeats across several decoder layers, with each layer refining the estimate.
  2. Geometry-Conditioned Depth Prediction: Predicting depth from a single image (monocular depth) is notoriously difficult. BoE makes this easier by providing the network with the 3D positions of other joints of the same person as context. If the model knows where the shoulder and elbow are in 3D, it has a much better "metric context" to guess the depth of the wrist.
  3. Target-Depth Masking (TDM): To prevent the network from simply "cheating" by copying its previous depth estimate, the authors use TDM. When predicting the depth of a specific joint, the model sees the 3D context of every other joint, forcing it to actually use the image data and the body's relative proportions.
  4. Learnable Depth-Aware Triangulation (LDAT): This is the core engine. Standard triangulation only uses 2D coordinates. LDAT incorporates the predicted depths as additional constraints in a differentiable algebraic system. This allows a single camera to contribute to the 3D reconstruction while still benefiting from the geometric precision of multiple views when they are available.
Figure 1. Overview of BoE. Starting from initial 3D pose queries, the decoder projects joints into calibrated camera views to ex- tract image evidence. Each view provides 2D joint and geometry- conditioned depth predictions, which are combined through LDAT to update the 3D poses. The updated poses serve as references for subsequent decoder layers, enabling iterative refinement.

Datasets

The model was primarily trained and evaluated on:

  • CMU Panoptic: A massive multi-view dataset. The authors specifically tested on the "CMU0" configuration and evaluated how well the model generalizes to different numbers of cameras (3 to 7) and entirely unseen camera arrangements (CMU1–CMU4).
  • Campus and Shelf: Standard benchmarks used to test "cross-scene" generalization (moving from the lab environment of CMU Panoptic to different outdoor/indoor settings).

Key Insights and Ideas

  • Body Geometry as a Prior: The most significant "aha" moment is that the human body itself provides the best context for depth. By conditioning depth prediction on the 3D geometry of the rest of the skeleton, the model resolves monocular ambiguities that would baffle a standard depth estimator.
  • Soft Constraints vs. Hard Logic: Instead of choosing between "pure triangulation" and "pure deep learning," LDAT treats predicted depth as a weighted constraint. If the depth prediction is confident, it guides the 3D position; if not, the model relies more on 2D intersections from other views.
  • Iterative Synergy: The 3D pose informs the depth prediction, and the depth prediction improves the 3D pose. This feedback loop allows the model to "bootstrap" itself into a highly accurate reconstruction.
Figure 2. BoE decoder architecture and geometry-conditioned depth estimation. (a) Each decoder layer extracts image evidence through projective attention and predicts per-view 2D joint locations, confidences, depths, and depth gates. Learnable depth-aware triangulation (LDAT) fuses these measurements to update the 3D poses, while separate branches update appearance features and predict pose scores. (b) Target-depth masking (TDM) constructs a separate joint sequence for each target joint, excluding only its own reference-depth embedding while retaining its image feature. (c) Pose-context modeling (PCM) processes the masked sequences using a shared encoder and selects the contextualized feature of each target joint for depth prediction.

Key Results

  • Sparse View Mastery: In the most difficult test case—using only three cameras—BoE outperformed the previous state-of-the-art (MV-SSM) by a massive 10.6 AP25 points.
  • State-of-the-Art Accuracy: On the standard CMU Panoptic benchmark, BoE achieved a new best-in-class performance with 93.7 AP25 and the lowest error (15.4 mm MPJPE).
  • Robust Generalization: The model showed superior performance when tested on camera setups it had never seen during training, proving that the geometric inductive bias of LDAT helps it adapt to new environments.
Qualitative results showing BoE (bottom) correctly reconstructing occluded joints that MVGFormer (top) misses.

Limitations and Future Work

  • Cross-Scene Depth Shifts: The model struggled when moving from CMU Panoptic to the Campus dataset. The authors found that the depth predictor is sensitive to the "depth distribution" of the training data (e.g., if people are usually 2-5 meters away in training, the model struggles when they are 15 meters away).
  • Total Occlusion: While BoE is great at handling joints seen in only one camera, it still cannot reconstruct a joint that is hidden from all cameras.
  • Future Directions: The authors suggest incorporating kinematic models (bone length constraints) or learned 3D pose priors to help "hallucinate" joints that are completely invisible to the entire camera system.

Summary updated

Abstract

Even in multi-camera setups, occlusions can leave individual body joints with only a single usable 2D observation. Such an observation defines a viewing ray but leaves the joint's depth undetermined. We introduce Body of Evidence (BoE), a multi-view 3D human pose estimator that predicts per-camera joint depths alongside 2D joint locations. To reduce monocular depth ambiguity, we propose geometry-conditioned depth prediction, which combines image features with previously reconstructed 3D body geometry as contextual depth cues. We further introduce learnable depth-aware triangulation (LDAT), which augments differentiable algebraic triangulation with predicted depth constraints, enabling reconstruction even from a single contributing camera while preserving the benefits of geometric triangulation when multiple views are available. The reconstructed 3D poses subsequently guide depth prediction in the next iteration, allowing depth estimates and 3D geometry to progressively refine each other. Experiments on CMU Panoptic demonstrate state-of-the-art in-domain performance and superior average performance across evaluated camera subsets and unseen camera arrangements. BoE is particularly effective under sparse camera coverage, outperforming the previous state of the art by 10.6 AP25 points in the challenging three-camera setting.
BibTeX
@misc{brorsson2026body,
  author = {Erik Brorsson and Kilian Freitag and Josef Bengtson and Kristofer Bengtsson and Lars Hammarstrand and Knut Åkesson and Fredrik Kahl},
  title = {{Body of Evidence: Triangulating 3D Human Pose with Geometry-Conditioned Depth}},
  year = {2026},
  eprint = {2611.21044},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21044},
}
Figure