Leave-One-View-Out Epipolar Self-Training for Multi-View 3D Pose

Multi-camera systems for 3D human pose are usually trained one of two ways. With motion-capture labels, which are expensive, studio-bound, and absent for settings like operating rooms. Or self-supervised, by using an off-the-shelf 2D keypoint detector's outputs as labels. The self-supervised route has a ceiling built in. When the training target is the detector's own guess, the 3D model can at best reproduce the detector's mistakes. When a joint is hidden in one camera, the detector hallucinates, triangulation is pulled off, and no later 3D module can recover information that the 2D stage already threw away. We propose to break this ceiling with two linked changes. First, train each camera's 2D prediction against a target built only from the other cameras. Second, give the 2D network the means to meet that target at test time by letting it look along the matching line in the other views. That line is the epipolar line: the path in a second image on which a pixel's true match must lie.

In two sentences: the label for camera i is the 3D point triangulated from all cameras except i, reprojected into i. That label's error does not depend on i's own occlusion or hallucination, so the student can learn to be more accurate than its teacher. A feature-fusion module samples the other views along epipolar lines, and concentrates those samples near the depth the current 3D estimate predicts. This lets the 2D network use what the other cameras can see when its own view is blocked.

Core Claim

Self-supervised multi-view pose methods inherit the 2D detector as an upper bound on accuracy, and this bound comes from the training target, not from model capacity. The bound can be exceeded when two conditions hold. The target for each view must be statistically decoupled from that view's own detection error, and the network must have, at inference, access to the cross-view evidence the target was built from. Neither condition alone is enough. Cross-view targets without cross-view input teach a single-view detector to guess occluded joints from body priors. Cross-view input trained on the detector's own pseudo-labels learns to copy the detector.

The leave-one-out target is defined as follows. For joint $j$, let $u_{kj}$ be view $k$'s predicted 2D location, $w_{kj}$ a confidence weight, and $\pi_k$ the projection into camera $k$. The target is

$$\hat X_j^{(-i)} = \arg\min_X \sum_{k \neq i} w_{kj}\, \rho\big(\|\pi_k(X) - u_{kj}\|\big), \qquad \mathcal{L}_i = \sum_j \big\| \pi_i(\hat X_j^{(-i)}) - u_{ij} \big\|,$$

where $\rho$ is a robust penalty and the target is computed with gradients stopped, from an exponential-moving-average copy of the network. Suppose per-view errors are roughly independent with similar spread. Then the triangulated target from $m-1$ views has error shrinking roughly like $1/\sqrt{m-1}$, which is the usual averaging argument. With five cameras, it is therefore expected to be more accurate than any single view's detection. The argument fails when the same joint is hidden in most cameras. In that case errors are correlated, and that regime is an explicit measurement in the experiments, not an assumption.

Relation to Prior Work

The work that motivates the gap is self-supervised multi-view pose. EpipolarPose triangulates pseudo-3D targets from a frozen 2D detector, which it freezes to avoid collapse. It acknowledges that detector errors corrupt the targets and cannot be corrected, because no gradient reaches the 2D stage. SelfPose3d reprojects predicted 3D poses onto 2D pseudo-labels and drops the worst-fitting view per person. That limits damage, but the target is still the detector's output. Its error also rises from 24.5 mm to 43.5 mm on CMU Panoptic when going from five to three views. DisPOSE fixes cross-view identity association with a diffusion model over assignment matrices. It still takes its 2D inputs from off-the-shelf detectors and reasons over joint tokens rather than image features. Iqbal et al. enforce multi-view consistency through Procrustes alignment of per-view 3D predictions, processing each view in isolation. Our proposal differs from all four in the same way: the 2D stage is trained, its targets exclude the view being trained, and it sees the other views through features.

The method we reuse is Epipolar Transformers. That work samples $K$ features along a pixel's epipolar line in a neighboring view and fuses them back into the 2D backbone with attention. It reaches 26.9 mm MPJPE on Human3.6M, but only with full 2D and 3D supervision. We keep its sampler and change two things: what it is trained against (no labels), and where along the line it samples. For the second change we borrow from DFA3D. DFA3D shows that features gathered along a camera ray become entangled across depths unless weighted by a depth distribution, and that depth-weighted bilinear sampling achieves this cheaply. We apply that idea to epipolar sampling. The depth weights come from the current 3D estimate's position along the ray, so 3D geometry steers where 2D features are read. This fusion of image appearance with a geometric depth prior is our cross-modal ingredient.

The constraint comes from collapse results. Chen et al. needed an adversarial pose prior because geometric self-consistency alone produced degenerate skeletons. EpipolarPose froze its detector for the same reason. A leave-one-out loss can also be satisfied by all views agreeing on a wrong but consistent answer. We therefore keep a down-weighted, robust anchor to the original detector's pseudo-labels, and test whether removing it causes drift.

Research Program

We will build a multi-view 2D pose network: an HRNet-style backbone with epipolar fusion blocks at intermediate resolution. It is initialized from the same off-the-shelf detector the baselines use. Training proceeds in rounds. An exponential-moving-average teacher produces per-view heatmaps and confidences. Leave-one-out targets are triangulated for every view. The student is trained on these targets plus the weak anchor loss, with the fusion module's depth weighting fed by the teacher's 3D estimate.

The 3D output is the confidence-weighted triangulation of the student's 2D predictions. This keeps the comparison focused on whether the 2D stage improved, rather than on a separate lifter.

The analysis has three parts. We measure ceiling-breaking directly: the student's 2D error against ground-truth 2D projections, minus the initial detector's error, split by how many views see each joint. We measure how this gain depends on view count. Finally, we test calibration robustness, because epipolar sampling assumes exact extrinsics. We perturb camera rotations by small angles at test time and compare fixed sampling with a variant that learns small per-sample offsets perpendicular to the line.

Baselines

The naive reference is the frozen off-the-shelf detector followed by confidence-weighted robust triangulation, with no training on the target data. Every method must beat this, and it measures how much any self-training adds.

The strongest self-supervised competitor with reported numbers is SelfPose3d. It reports 24.5 mm MPJPE on CMU Panoptic with five views and 43.5 mm with three. We will run it from released code with the same 2D detector initialization, the same training sequences, and a matched GPU-hour budget, and tune both methods with the same small validation sweep. Beating it means lower MPJPE at five views and a clearly smaller gap at three views. The three-view setting is where our cross-view input should matter most. DisPOSE is reported to outperform SelfPose3d on Panoptic and on the operating-room benchmark MM-OR, but no reported numeric result was found on Lacuna for DisPOSE on these benchmarks. We will run it if code is available and otherwise state the omission.

As references, we include the supervised VoxelPose figure of 17.7 mm on Panoptic quoted by SelfPose3d. On Human3.6M we include the supervised Epipolar Transformers result of 26.9 mm MPJPE and the self-supervised EpipolarPose result of 60.56 mm. EpipolarPose is a monocular method, so it is a loose reference rather than a like-for-like comparison.

Two ablations isolate the key ingredients. The first is our epipolar-fusion network trained on the detector's own pseudo-labels rather than leave-one-out targets, which tests whether the target matters. The second is a single-view network with no fusion trained on leave-one-out targets, which tests whether cross-view input matters. A third ablation replaces depth-weighted sampling with uniform sampling along the line.

Why not just…?

Why not just self-train the single-view detector on triangulated pseudo-labels? This is the most serious threat. Iterated triangulate-then-retrain may capture most of the gain, because it also produces targets that average over views. It is exactly our second ablation, run with the same rounds and budget. Our prediction is that it improves joints visible in most views but not joints hidden in the view being evaluated. In that case the single-view network can only guess from body shape, while the fused network can read the other cameras. If the ablation matches the full method on the hardest view-count bins, the fusion module adds nothing, and we will report the simpler method as the result.

Why not just drop unreliable views at triangulation, as SelfPose3d does? Discarding the worst view removes a bad measurement but never corrects it. The final estimate is then built from fewer cameras, which is consistent with SelfPose3d's sharp degradation at three views. The per-view comparison against ground-truth 2D will show whether our corrected views help, rather than merely staying out of the way.

Why not just train Epipolar Transformers with full supervision on Human3.6M and transfer? The method needs 3D labels, which the target settings lack. Transfer across camera layouts and scenes is untested in the file. We will report a Human3.6M-trained model evaluated on Panoptic and Shelf/Campus as an additional reference. If it beats our label-free result on transfer, the case for self-training in new rigs weakens accordingly.

Proposed Experiments

The primary benchmark is CMU Panoptic at five and three views, reporting MPJPE and the $AP_{25}$ metric used by SelfPose3d. Human3.6M serves as a second benchmark with four views and no 3D labels used, compared against the supervised 26.9 mm reference. MM-OR POSE is the heavy-occlusion stress test, where correlated cross-view occlusion should limit our method. We include cross-scene transfer from Panoptic to Shelf and Campus.

The decisive check is the per-joint 2D error difference between student and initial detector, binned by number of views in which the joint is visible. The claim holds if the student is strictly better than the teacher for joints occluded in the evaluated view but visible in at least two others. It must hold under the full method and fail, or largely shrink, under both ablations. The claim is falsified if gains appear only for joints the detector already sees well, or if the anchor-free variant drifts while the anchored one merely matches SelfPose3d.

A secondary check perturbs extrinsics by 0.5 to 2 degrees and reports MPJPE degradation for fixed versus offset-learning epipolar sampling. This quantifies the calibration fragility that both rectification-based matching and projection-based fusion share.