Peer-Relative Uncertainty for Active Monocular 3D Detection

A single camera cannot measure distance directly. A small car in the image might be a normal car far away or a toy close up, so a detector that predicts 3D boxes from one image is solving a problem with no unique answer. A model that reports its own uncertainty will therefore be most unsure about distant objects, and that unsureness grows roughly with the square of distance whether or not the object is anything new. This matters for active learning, the practice of letting the model pick which unlabeled examples a human should annotate next. "Annotate what the model is unsure about" turns into "annotate tiny objects near the horizon." IDEAL-M3D, an instance-level active learner for monocular 3D detection, documents this depth-ambiguity bias and responds by dropping uncertainty altogether in favor of diversity sampling.

We propose to keep uncertainty but judge each object only against its peers. Peers are objects at a similar distance, size, truncation level and appearance, and the peer groups are discovered automatically by a Dirichlet-process clustering. This is a clustering method that decides for itself how many groups the data supports and opens a new group when an object fits none of the existing ones. Within each group we track how fast the detector's real depth error falls as labels arrive, across active-learning rounds. An object is worth labeling if it is unusually uncertain for its group and its group is still improving, or if it starts a group the labeled set has never covered. Distance-driven ambiguity that no amount of labeling will fix shows up as a group whose error has stopped falling, and it gets deprioritized.

Core Claim

The bias IDEAL-M3D found does not show that uncertainty is useless for monocular 3D active learning. It shows that raw uncertainty mixes two things. One is projection variance, which is predictable from geometry. The other is genuine novelty. We claim these can be separated without LiDAR and without a hand-written model of the noise. The separation comes from two signals: how an object's uncertainty compares with that of objects in its own nonparametrically discovered group, and whether that group's labeled error has been shrinking. We expect the resulting selector to reach full-data accuracy on KITTI with fewer labeled boxes than IDEAL-M3D's diversity selector. We also expect it to choose a distance distribution that is not skewed toward the horizon.

How the Selector Works

The detector is GUPNet++. It models 2D and 3D object height as Laplace distributions and pushes that spread through the pinhole formula $Z = H_{3d} f / h_{2d}$ (with $Z$ the depth, $H_{3d}$ the physical height, $f$ the focal length and $h_{2d}$ the image height) to get a depth distribution for each box. Its depth standard deviation $\sigma_i$ for object $i$ is our raw uncertainty. The detector supplies the uncertainty; the selection rule is ours. GUPNet++ and its predecessor GUP Net train passively and never use this quantity to choose data.

Each detected object in the labeled and unlabeled pools gets a descriptor. The descriptor joins two kinds of input. The first is features taken just before the 3D heads, which IDEAL-M3D found far better than generic features for its Core-Set Box3D variant. The second is explicit geometric attributes: log predicted depth, $h_{2d}$, truncation and occlusion flags, and class. DDFH showed in LiDAR active learning that adding geometry to deep features helps, but it fit a finite Gaussian mixture whose number of components is fixed in advance.

We cluster the descriptors with DP-space. DP-space is the fast deterministic limit of a Dirichlet-process mixture of PCAs. It finds both the number of groups and each group's dimensionality, and on Hopkins 155 it ran in about 2 seconds against more than 500 seconds for sparse subspace clustering. That speed is what lets us re-cluster the whole pool every round. Its known weakness is a penalty $\lambda$ that sets how readily a new group is opened. We set $\lambda$ from a quantile of nearest-neighbor descriptor distances on the initial labeled seed, and we report sensitivity across a range of $\lambda$ values rather than hiding it.

For object $i$ in group $k$ at round $t$, the score is

$$s_i = \frac{\sigma_i}{\bar\sigma_k}\,\max\!\big(0,\; e_k^{(t-1)} - e_k^{(t)}\big)^{\gamma},$$

where $\bar\sigma_k$ is the median predicted depth spread among group-$k$ members, and $e_k^{(t)}$ is the mean absolute depth error on held-out labeled members of group $k$ after round $t$. The exponent $\gamma \ge 0$ controls how strongly learning progress counts. Setting $\gamma = 0$ gives pure peer-relative uncertainty. Using the median follows Peng et al., who found that selecting at maximum aleatoric uncertainty picks up unlearnable points, while median-based selection avoids them.

Some groups have no labeled members. These are new regions of descriptor space, and they bypass the progress term. Each round, a fixed fraction of the budget is spent on the medoids of such groups.

Annotation is per box, not per frame. We reuse IDEAL-M3D's masked instance loss so that unlabeled objects in a partly labeled image do not contribute training signal.

Research Program

The work has three parts. First, a diagnostic study on KITTI. Here we measure how much of GUPNet++'s per-object $\sigma_i$ on the validation set is explained by depth alone, by depth plus truncation, occlusion and class, and by DP-space group membership. If depth explains nearly all of it, the clustering adds nothing and the paper's claim narrows. Second, the full active-learning loop with the score above. Third, ablations that remove one ingredient at a time. On one ablation we will be honest in advance: if fixed distance bins work as well as learned groups, the result becomes "depth-normalized uncertainty suffices." That would still be a useful correction to IDEAL-M3D, but it would be a smaller contribution.

Baselines

The naive reference points are random instance selection and raw GUPNet++ $\sigma_i$ ranking. The raw ranking should reproduce the horizon bias, and confirming that it does is a sanity check on our setup.

The strongest existing method is IDEAL-M3D. It uses a diverse three-backbone ensemble with Core-Set selection on pre-head features, and it reports reaching 100% of full-data performance with 60% of the labeled data on KITTI validation. On Rope3D it reports over 97% accuracy with 25% of the data for cars. We run it with our GUPNet++ detector and with its own ensemble. All methods share the same seed set, per-round box budget, training schedule and compute accounting, and we count IDEAL-M3D's reported 85% extra training time as part of its cost.

Beating IDEAL-M3D means reaching its full-data-equivalent moderate-Car AP40 with clearly fewer boxes than its 60%, averaged over at least three seeds. A tie at lower selection cost would be a weaker but still reportable result.

MonoLiG reports a 17% better labeling-cost saving rate on KITTI. It needs a LiDAR teacher, so we report it only as an upper reference that has access to an extra sensor, not as a like-for-like competitor.

The ablations isolate each ingredient. Peer normalization with $\gamma = 0$ isolates the role of learning progress. Replacing DP-space with k-means at the best K found by grid search over a held-out round isolates the nonparametric clustering. Replacing groups with fixed distance bins isolates whether anything beyond distance matters. Removing the new-group budget isolates the role of novelty.

Why not just…?

Why not just divide uncertainty by depth squared, or rank within distance bins? This is the most serious threat, because the pinhole formula predicts exactly that scaling. Our counter-argument is that ambiguity also depends on truncation and occlusion. GUP Net and GUPNet++ both note that the projection formula fails when $h_{2d}$ cannot be measured, and a distance-only correction cannot see these cases. The diagnostic study and the distance-bin ablation settle the question directly. If the bins match our selector, we will say so.

Why not just use Core-Set diversity as IDEAL-M3D does? Diversity treats every region of feature space as equally worth covering. It cannot tell whether the detector is already good in a region or whether a region is hopeless. Our progress term supplies both pieces of information. A head-to-head comparison at equal budget and with the same detector shows whether that information pays off.

Why not a finite mixture or k-means with tuned K, as in DDFH or CRB? The pool and the feature space shift every round, so a K tuned once goes stale. DP-space also showed that finite mixtures of PCAs degrade when cluster dimensions are misspecified. Still, a per-round grid search over K is a fair competitor. We run it, and we report how much extra compute that tuning costs.

Why not ensemble disagreement as the epistemic signal? Ensemble members trained on the same ill-posed geometry tend to disagree most on the same distant objects, so disagreement probably inherits the horizon bias. We include IDEAL-M3D's ensemble variance as a scoring rule to test this.

Proposed Experiments

The primary benchmark is KITTI 3D detection: AP40 for Car, Pedestrian and Cyclist on validation, plotted against the fraction of labeled boxes over five to eight rounds. Secondary benchmarks are Waymo, where IDEAL-M3D reports over 95% accuracy with 25% of the data, and Rope3D, which tests whether peer groups adapt to elevated roadside viewpoints.

The decisive checks are three. The first is the depth histogram of selected boxes compared with the pool's distribution. The second is per-group error curves, which show whether deprioritized groups truly stop improving when they are labeled anyway; for this we run a held-out probe that labels a sample from them. The third is the head-to-head against both IDEAL-M3D and the distance-bin ablation at matched budgets.

The proposal fails if the distance bins tie our selector, or if the gains disappear across $\lambda$ settings.