Keeping the Flip: Multi-Hypothesis Geodesic Self-Training for Synthetic-to-Real Pose

A network trained on rendered images to predict 3D object orientation usually fails on real photographs in a specific way. It does not drift a few degrees. It often lands on a pose that is nearly the right one turned upside down or rotated 180° about the object's long axis, and it is confident about it. The standard way to adapt such a network to real images is self-training: the network labels unlabeled real photos with its own predictions (pseudo-labels) and then trains on them. A geodesic loss measures error as the true rotation angle between two orientations, and it has become the preferred rotation loss because its error signal does not fade when the prediction is far from the target. That same property becomes a problem during self-training. When a pseudo-label is flipped by about 180°, the loss pushes hardest in the wrong direction. Euclidean or chordal losses saturate near 180°, so they quietly ignore these examples. That avoids the damage, but it also throws away what the flipped example still tells us: the object is at one of two specific orientations.

This proposal makes a simple claim. Synthetic-to-real rotation errors are structured, concentrating on a few discrete "confusion" rotations, and the right response is to represent that ambiguity rather than penalize or discard it. The network predicts a small set of rotation hypotheses with scores. On real images, the teacher's set of high-scoring hypotheses becomes a set-valued pseudo-label. The student is trained with a geodesic matching loss between its hypothesis set and the teacher's set, with gradients computed in the tangent space of the rotation group. Each real image then contributes accurate local geometry around every plausible mode. Choosing among modes is learned mainly where it can be learned reliably, which is on synthetic data with exact labels.

What is known, and what is missing

Geodesic losses are well established for supervised rotation regression. 3D Pose Regression Using Convolutional Neural Networks regresses continuous pose on PASCAL3D+ with a geodesic objective. A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Images combines orientation bins with geodesic refinement and reaches 10.10° median error. The same authors observe that continuous geodesic regression collapses toward a mean pose on symmetric or ambiguous objects; this motivates the multi-hypothesis head. Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration shows that geodesic loss beats mean squared error for large misalignments. It handles the instability of the arccos formula with a two-stage schedule, and its domain transfer is between MRI modalities, not from graphics to photographs. Tangent Space Backpropagation for 3D Transformation Groups supplies the method we reuse: closed-form Jacobians in the rotation group's tangent space, which remove the NaN gradients at zero rotation. That work targets SLAM, not single-image adaptation.

On the transfer side, three lines of work adapt pose estimators from synthetic to real data. All of them either avoid geodesic regression or rely on extra views. Robust Category-Level 3D Pose Estimation from Synthetic Data builds the SyntheticP3D dataset and adapts with contrastive features plus pseudo-labels, estimating pose by iterative inverse rendering. ONDA-Pose cleans pseudo-labels using multi-view Structure-from-Motion. Synthetic-to-Real Pose Estimation with Geometric Reconstruction names the "confidently wrong" pseudo-label problem and filters labels with kinematic checks and video-pair warping on Euclidean keypoints. None of them asks how the shape of the rotation loss at large angles interacts with pseudo-label noise. None of them keeps several orientations alive in a single-view, feed-forward estimator. When Regression Meets Manifold Learning uses angle-aware metric learning on synthetic depth, but it regresses quaternions with plain Euclidean loss and requires real depth labels.

Two further papers constrain the design. Deep Orientation Uncertainty Learning based on a Bingham Loss models orientation uncertainty with a distribution on the rotation group. It is the closest probabilistic competitor and must appear as a baseline. Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation shows that much of the visual gap is high-frequency image statistics, which Amplitude Spectrum Augmentation (ASA) closes. We include ASA in every source-domain model so that the loss is not credited with gains that cheap augmentation already delivers.

Core Claim

The claim has two parts. First, under synthetic-to-real shift, the angular error of a teacher's top-1 rotation pseudo-labels is bimodal: one mass sits near zero and a second sits near a few object-specific confusion rotations, often near 180°. Second, a student trained to cover the teacher's full hypothesis set with a geodesic matching loss adapts better than any scalar loss applied to the top-1 label, whether geodesic, chordal, truncated, or confidence-filtered.

Concretely, the network outputs $K$ rotations $R_1(x),\dots,R_K(x)$ with scores $s_k(x)$. On a real image, let the teacher's pseudo-label set $\mathcal{T}(x)=\{\hat R_j\}$ be its hypotheses with score above a threshold $\tau$. The real-image loss is

$$L_{\text{real}}(x)=\sum_{\hat R_j\in\mathcal{T}(x)} \min_{k}\, d\big(R_k(x),\hat R_j\big), \qquad d(R,\hat R)=\big\|\mathrm{Log}(R\hat R^\top)\big\|,$$

where $\mathrm{Log}$ maps a rotation to its axis-angle vector, so $d$ is the geodesic angle. Gradients are taken through the tangent-space Jacobians of Tangent Space Backpropagation for 3D Transformation Groups. This loss never pulls a hypothesis across the group toward a flipped label. Each teacher mode is matched by the nearest student hypothesis, so a flipped label only refines a nearby hypothesis.

The scores are trained differently. On synthetic images, they are trained with cross-entropy toward the hypothesis nearest the true pose. On real images, they are trained only weakly, by distillation from the teacher's softened scores. The design rests on a deliberate assumption: real images are informative about which orientations are plausible and how precise each is, while synthetic labels are what decide among plausible modes. If that assumption fails, the decisive check below will show it.

Research Program

The work begins with diagnosis. For a source-only model, we histogram the angular error of top-1 predictions on labeled real validation images. We use two settings. The first is category-level: SyntheticP3D to PASCAL3D+. The second is instance-level: BlenderProc PBR training images from the BOP Challenge 2020 to single real frames of LineMOD and Occluded-LineMOD, with no multi-view information used.

We then test whether the second error mode lies near a small set of rotations. We cluster the relative rotations $\hat R\,R_{\text{gt}}^\top$ for wrong predictions. We also measure how often the true pose appears among the teacher's top-$K$ hypotheses, which is the ceiling on what set-valued pseudo-labels can recover. If errors are diffuse rather than clustered, the multi-hypothesis argument weakens. In that case we would report that loss-tail shape is what matters and recommend the simplest bounded loss.

The method itself is a standard teacher–student loop. The backbone is shared across all variants. The teacher is updated as an exponential moving average of the student. The student is trained on synthetic data with a best-of-$K$ geodesic loss plus score cross-entropy, and on real data with $L_{\text{real}}$. We vary $K$ (1, 2, 4, 8) and $\tau$. We also log, per real image, whether the teacher's top-1 was flipped, which lets us measure what happens to exactly the examples that break single-label self-training.

Baselines

The naive reference point is a source-only model. It is trained on synthetic data with ASA and a single-hypothesis geodesic loss, then evaluated zero-shot on real images. We found no reported result on Lacuna for synthetic-only zero-shot accuracy on PASCAL3D+ or single-view LM-O, so we will measure it ourselves. Every adapted method must beat it to count.

The direct competitor for the loss question is single-label self-training. It uses the same teacher–student loop and the same images and compute, with only the real-image loss swapped. We run four variants: full geodesic, chordal (which saturates near 180°), geodesic truncated at an angle chosen on a synthetic validation split, and geodesic with confidence filtering. Each variant gets the same hyperparameter search budget as ours, selected on a small labeled real validation split that is shared by all methods. Our method beats this family if it improves median error and $Acc_{\pi/18}$ over the best of the four variants on both benchmarks, and specifically on the subset of images where the teacher's initial top-1 was flipped.

The probabilistic competitor is a Bingham-loss model (Deep Orientation Uncertainty Learning based on a Bingham Loss), self-trained by weighting pseudo-labels with predicted concentration. It is the strongest alternative way to express ambiguity without multiple heads.

The discrete competitor is bin-plus-delta classification-regression (A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Images), self-trained with soft bin pseudo-labels. Cross-entropy over bins already handles multimodality. For reference, that paper reports 10.10° median error and 0.8588 accuracy on PASCAL3D+ with full real supervision, which is an upper anchor rather than a fair target.

The strongest existing synthetic-to-real method on the category-level task is CC3D (Robust Category-Level 3D Pose Estimation from Synthetic Data). It reports matching full-data methods with 10% of real labels and a 10.4% gain at $\pi/18$ with 50% of real labels. We will run it under the same real-label budgets (0%, 10%) and also report inference time, since it uses iterative inverse rendering. On LM-O, the reference is ONDA-Pose at 69.3% ADD-S (93.2% on LineMOD). That method uses multi-view SfM, so we treat it as an upper bound with stronger supervision. Approaching it from single frames would be the meaningful result.

Two ablations isolate the key ingredient. In the first, a multi-hypothesis head is trained on real images against only the teacher's top-1 label, which tests whether the set-valued pseudo-label matters. In the second, $K=1$ is used with the set-valued pseudo-label, which collapses to a min over teacher modes and tests whether the student needs its own multiple hypotheses.

Why not just…?

Why not just use the chordal loss, which already saturates near 180°? This is the most serious threat. The chordal loss's gradient goes to zero for flipped labels, so it removes their harm. It also removes their information: a flipped example still constrains the object to two orientations and still carries fine local geometry around each. Our method should therefore win specifically on images whose initial teacher label was flipped, measured by the student's best-of-$K$ and top-1 accuracy on that subset. If chordal self-training matches us there, the honest conclusion is that tail saturation is all that is needed, and we will report it that way.

Why not just filter pseudo-labels by confidence? Synthetic-to-Real Pose Estimation with Geometric Reconstruction reports that under domain shift, wrong pseudo-labels are often confident, which is why it needed kinematic checks. The diagnostic histogram directly measures whether teacher confidence separates the near-zero and flipped modes. If it does separate them, filtering is a strong and cheap baseline, and it is included at an equal tuning budget.

Why not just discretize orientation and use cross-entropy? Bins handle multimodality for free, as in Domain Randomization for Scene-Specific Car Detection and Pose Estimation. However, the Mahendran papers show that pure binning loses sub-bin precision, which matters at $\pi/18$. The bin-plus-delta baseline tests whether a discrete mode representation with continuous refinement already captures our gain. If it matches us, the contribution reduces to showing that set-valued self-training matters, not that the specific geodesic set loss does.

Why not just render more photorealistic data? We cannot test unlimited rendering, but we can test the cheap version. Every model includes ASA, and on LM-O we also train on the full BOP PBR set. If adaptation adds nothing over a source-only model on that data, the loss question is moot.

Proposed Experiments

The first decisive check is the error histogram. If the teacher's top-1 errors on real validation images show a clustered second mode, and the true pose lies in the teacher's top-$K$ set substantially more often than in its top-1, the premise holds.

The second decisive check is on the flipped subset. On real images where the initial teacher was flipped, after adaptation, our student's top-1 $Acc_{\pi/18}$ must exceed that of the best single-label variant, and its best-of-$K$ accuracy must exceed its top-1 accuracy by less than before adaptation, meaning the right mode is being promoted.

The third check covers the full benchmarks. We report median error, $Acc_{\pi/6}$ and $Acc_{\pi/18}$ on PASCAL3D+, and ADD-S on LineMOD and LM-O, with three seeds each. We expect the method to help most on categories and objects with near-symmetries, such as boats, tables, and LM-O's eggbox and glue. It should do no harm on clearly asymmetric objects. If it does harm, that indicates the extra hypotheses are absorbing real errors.

Finally, we compare tangent-space gradients with arccos autodiff to confirm that training remains stable without a mean-squared-error warm-up. This answers the two-stage schedule of Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration.