Querying Labels Where Models Disagree About the Modes, Not Where Poses Spread

A camera localizer predicts where a camera is from a single image. In buildings full of identical staircases, repeated ceiling panels or rows of matching chairs, one image can come from several places. This is called perceptual aliasing: different places produce nearly the same picture. A good localizer should then predict several candidate poses at once. Separately, active learning is the practice of choosing which images to pay to label. Its most common rule is "label the image the model is least sure about," usually measured by predictive variance, meaning how widely the model's predictions scatter. These two ideas conflict. In an aliased scene, the model's predictions scatter across all the look-alike locations no matter how many labels it has seen. A variance-driven learner will therefore keep buying labels for images that can never be made unambiguous.

This note proposes a different acquisition rule for training pose regressors in repetitive environments. The rule separates two kinds of spread. The first is spread within each model's multimodal prediction, which reflects the ambiguity that is really in the scene. The second is disagreement between independently trained models about which modes exist and where they sit, which reflects missing training data. Labels are spent only on the second. The claim can be tested directly. On scenes with known alias structure, a learner that acquires by total variance should keep drawing labels from structurally ambiguous frames without improving. A learner that acquires by between-model disagreement should reach full-data accuracy with a fraction of the pose labels.

Core Claim

The active learning literature treats predictive variance as a stand-in for epistemic uncertainty, the kind that more data removes. That assumption underlies Dropout-based Active Learning for Regression, which ranks unlabeled inputs by the spread of repeated stochastic forward passes, and the variance-reduction results in Active learning for logistic regression: an evaluation. The aliasing literature shows why the assumption fails for localization. A Probabilistic Framework for Visual Localization in Ambiguous Scenes (VAPOR) shows that a well-trained model should spread its output across up to six physically distinct poses for a single ceiling image. For that image, the spread is correct and permanent.

The fix is to measure disagreement over whole predictive distributions rather than over point predictions. Train $K$ multimodal pose regressors. Each is a VAPOR-style network that outputs a cloud of $SE(3)$ pose samples for image $x$. Write $p_k(y\mid x)$ for the pose distribution of member $k$. Total spread splits into two terms:

$$ \underbrace{\tfrac{1}{K}\textstyle\sum_k D\big(p_k,\ \bar p\big)}_{\text{between-member disagreement (acquire on this)}} \quad\text{vs.}\quad \underbrace{\tfrac{1}{K}\textstyle\sum_k S(p_k)}_{\text{within-member spread (aliasing; ignore)}}. $$

Here $\bar p$ is the average of the members' distributions. $D$ is a sample-based distance between pose distributions; we use maximum mean discrepancy, which compares two sample sets through a kernel, applied to translation and a geodesic rotation distance. $S$ is the analogous self-spread of one member's samples. This is the law-of-total-variance idea behind disagreement-based acquisition, written for sample clouds on $SE(3)$. On that space, closed-form entropies of mixtures are not available, and a naive variance averages modes several meters apart into one large number. The new part is not the decomposition itself. It is showing, on scenes with known alias structure, that the decomposition is what keeps active learning from failing on aliased images, and that it transfers to pose regression.

Positioning Against Prior Work

The closest bridging work is Safety-aware Active Learning with Perceptual Ambiguity and Criticality Assessment. It argues that acquisition should account for ambiguity, using human-annotator ambiguity ratings on traffic-light classification, and it reports no experiments. We keep its motivation but replace subjective ratings with a score computed from the models, set in continuous 6-degree-of-freedom regression, and tested quantitatively.

Active Neural Localization also breaks aliasing "actively," but the agent moves its body toward distinctive landmarks. Its perception network is trained offline, and it never chooses which data to label.

VAPOR and Multi-output Learning for Camera Relocalization supply the method: both show how to keep several pose hypotheses alive, through a winners-take-all loss or a boosted portfolio of predictors. Both assume the training set is fixed and fully labeled.

Two papers constrain the design. Active Bias shows that picking samples by high loss chases unlearnable points, while variance across training iterations picks learnable ones. Our argument extends this: under aliasing, even variance picks unlearnable points unless it is decomposed. Deep Active Learning with Manifold-preserving Trajectory Sampling warns that greedy uncertainty sampling skews the labeled pool away from the data distribution. For that reason, a diversity term is included as a measured ablation rather than assumed away.

Research Program

The base model is a VAPOR-style regressor: a ResNet-18 encoder, a latent sampler and a pose decoder trained with the adaptive winners-take-all loss. It is instantiated $K=5$ times with different seeds and bootstrap resamples of the labeled pool. VAPOR reports 2.26 ms for 1,000 pose samples on one GPU, so scoring a pool of tens of thousands of frames with 5 members and a few hundred samples each is cheap compared with retraining.

Active learning runs in rounds. Each round starts from a small random seed set, scores every unlabeled frame by between-member disagreement, acquires a batch, retrains all members, and repeats. The pose oracle is simulated from each dataset's existing ground truth, so label cost is counted in frames.

Two analyses accompany the learning curves. The first is a saturation test. We add many labels at known aliased locations, such as the Ceiling dataset panels whose six equivalent poses are known, and track both terms of the decomposition. The prediction is that within-member spread stays high while between-member disagreement falls toward zero. If disagreement also stays high, the members are not converging on the alias set, and the method's premise fails.

The second analysis is an acquisition audit. For each method, we measure the fraction of queried frames that fall in known-aliased regions and plot it against the accuracy gain per round. Frames in transition regions, where alias modes diverge and the members disagree about mode placement, are where we expect the useful labels to be.

Baselines

Random acquisition. The naive reference point is the same five-member VAPOR ensemble trained on uniformly random frames, at matched label budgets, with identical architecture, epochs and hyperparameter search. For the full-data ceiling, VAPOR reports recall of 0.99 at 0.2 m / 15° on the Blue Chairs scene of the Ambiguous Relocalization Dataset (Zangeneh et al.). Bingham mixture density networks reach 0.83 and PoseNet 0.40 in the same paper. Beating random means reaching at least 0.95 of the ensemble's own full-data recall with substantially fewer labels: we set the bar at half as many labels or fewer, averaged over five seeds, with non-overlapping confidence intervals.

Total-variance acquisition. The strongest existing competitor is uncertainty acquisition in the style of Dropout-based Active Learning for Regression, applied in two ways. The first applies it to the pooled pose samples of the same ensemble, as total spread. The second is the original form: a unimodal pose regressor with 25 dropout passes. It is the most direct representative of "query where predictions vary," and it is the method the core claim says should fail on aliased frames. It gets the same models, pool, batch sizes and retraining budget. Its dropout rate is tuned over the same grid we use for our method's kernel bandwidth. Beating it means higher recall at every budget beyond the seed set on the Ceiling dataset and on Blue Chairs, and a clearly lower share of queries spent in aliased regions.

Isolating the ingredient. The key ablation compares three scores computed from the same five members: total spread, within-member spread alone, and between-member disagreement alone. Any gain then comes from the decomposition, not from the ensemble.

Diversity baselines. A greedy max-min diversity baseline, which picks frames far apart in feature space, and a spatial-coverage baseline, which picks frames far apart in camera position, test whether coverage alone suffices. A variant that adds the labeled-versus-unlabeled distribution-matching term from MPTS tests whether manifold skew hurts our score.

Why not just…?

Why not just train VAPOR on randomly sampled frames? VAPOR already handles aliasing at test time, so this is a serious threat. The question is purely about label efficiency. In repetitive scenes, random sampling spends most of its budget on redundant corridor or panel views. The file contains no study of VAPOR's accuracy as a function of the number of training labels, so we do not know how steep that curve is. If random sampling reaches 0.95 of full-data recall within about 20% of the labels, active learning has little to offer here, and the random baseline's learning curve will show it.

Why not just select by highest loss or hard examples? Active Bias reports hard-example mining at about 0.92% error on noisy MNIST against about 0.61% for variance-based selection, because loss targets points that cannot be learned. In our setting, a single-pose loss is large exactly on aliased frames, where the ground-truth pose is only one of several valid answers. We include max-loss as a baseline and expect it to fail the acquisition audit more badly than total variance does.

Why not just maximize spatial or feature coverage? Coverage ignores what the model already knows. Dropout-based Active Learning for Regression reports up to 20% improvement of uncertainty over greedy max-min, and MPTS reports trajectory-based variance beating CoreSet and BADGE. In aliased scenes, however, coverage has one real advantage: it does not over-sample the ambiguous frames. It could therefore beat total variance while losing to our score. That ordering is the one the experiment is designed to distinguish.

Why not just use longer image sequences at test time? Longer sequences address inference, not label cost, so they are complementary rather than competing. TOPO-Bench also reports that sequence matching is fragile: long windows break under turns, and short ones fall back to ambiguous matching. SG2Loc reports about 2.9 s per frame at 25 frames. We do not test this alternative head-to-head. A better-trained single-frame multimodal model should also give any sequential filter better inputs.

Proposed Experiments

The primary benchmarks are the Ambiguous Relocalization Dataset scenes (Blue Chairs and the Ceiling dataset) from Zangeneh et al., where the alias structure is known, and 7-Scenes Stairs, where Guzmán-Rivera et al. report about 20% improvement from multi-output prediction over a single-output baseline, indicating strong ambiguity. The other 7-Scenes rooms serve as low-ambiguity controls. On these, our score and total variance should behave almost identically, since there is little within-model spread to remove.

The metrics are three: recall at 0.2 m / 15° as defined by VAPOR, plotted against labeled-frame count; labels needed to reach 0.95 of full-data recall; and, on the Ceiling dataset, mode coverage, meaning the fraction of the six true locations recovered.

The result that would settle the question has three parts. The first two come from the saturation test and the audit: total-variance acquisition should put a large share of its queries on Ceiling-panel frames without any gain in recall, while between-member disagreement shifts toward transition frames and falls on saturated aliased regions. The third is agreement across methods: our score should beat random, total variance and coverage on the ambiguous scenes, and tie them on the control rooms. If the control rooms also show a gap, the gain is coming from something other than handling aliasing, and the claim is wrong as stated.

A secondary extension, run only if the pose results hold, would carry the same decomposition to place recognition on TOPO-Bench. There, the best reported Balanced Localization Accuracy is 0.295, from Probabilistic Belief Update. The extension would test whether acquiring on disagreement raises accuracy on the ambiguous-plus-true-revisit category without lowering it on the ambiguous-only category.