Choosing the First Frames to Label by How Sensors See, Not Where the Car Was

Active learning is a way to cut annotation cost: the model looks at a pool of unlabeled data and asks a human to label only the most useful items. It fails at the very start, a problem called the cold start. With almost no labels, the model's sense of what is "useful" is noise, and it often does worse than picking at random. For 3D object detection in autonomous driving, the working fix is to ignore the model at first and choose frames that are spread out in space and time, using the car's GPS log. This treats space as isotropic, meaning every direction is equally informative. Two frames taken at the same spot count as redundant even if one faces north and the other south. It also ignores the fact that the car's sensors are themselves strongly direction-dependent. A LiDAR (a laser range scanner) measures distance along its beam precisely but is blurrier across the beam. A radar is precise in range but very poor in bearing. A camera is precise sideways but poor in depth.

This note proposes a cold-start selection rule for driving data that needs no trained model and is built on this directional sensor physics. Each candidate frame is summarized by what it would contribute to observing the drivable world: for every map cell in view, a small ellipse of measurement precision whose shape comes from the sensor's noise model. The initial label set is chosen to maximize the accumulated, direction-aware precision over the map. Because precision ellipses from different viewing angles combine, the rule rewards seeing the same intersection from crossing directions and penalizes seeing it repeatedly along the same line of sight. A distance-based diversity rule cannot express that distinction.

Core Claim

Cold-start frame selection for 3D detection should maximize anisotropic, multi-view observation coverage of the scene, computed from known sensor noise models and ego poses, rather than isotropic geographic or feature-space distance. The improvement should grow with how anisotropic the sensor suite is. It should be largest for radar-camera detectors, smaller for LiDAR-only detectors, and near zero if the noise model is replaced by a round one.

The objective is concrete. For frame $f$ and map cell $c$ within that frame's sensor range, let $\Lambda_{f,c} = R_{f,c}\,\mathrm{diag}(\sigma_{r}^{-2}, \sigma_{t}^{-2})\,R_{f,c}^{T}$ be a 2×2 bird's-eye precision matrix. Here $R_{f,c}$ rotates into the ray direction from the sensor to the cell, $\sigma_r$ is the range standard deviation, and $\sigma_t$ is the transverse standard deviation. For LiDAR, $\sigma_t$ grows with range times beam divergence, as in the beam model of Park et al.. For radar, it grows with range times angular resolution. For the camera, depth error grows roughly quadratically with distance. A selected set $S$ is scored by

$$F(S) = \sum_{c} w_c \,\log\det\Big(I + \Lambda_0^{-1/2}\big(\textstyle\sum_{f\in S}\Lambda_{f,c}\big)\Lambda_0^{-1/2}\Big),$$

where $w_c$ restricts the sum to drivable-area cells from the nuScenes HD map, and $\Lambda_0$ is a reference precision that sets the scale. The log-determinant of a sum of positive semidefinite matrices is monotone and submodular in $S$. That property is what lets greedy selection come within $1-1/e$ of the best possible set, and it is the reason for choosing this form. The determinant is where anisotropy matters. Two frames that see a cell along perpendicular rays give a nearly round, large combined ellipse. Two frames along the same ray only make one axis longer, and the log-determinant saturates.

Research Program

The motivation comes from two places. Maier-Hein et al. show that when noise is direction-dependent, even a basic geometric step goes wrong if it uses Euclidean distance: switching to a covariance-weighted correspondence raised the correct match rate from 54% to 87% at σ = 9 mm. Exploring Diversity-based Active Learning for 3D Object Detection solves the cold start on nuScenes with a GPS graph and timestamp diversity, but it treats location as isotropic and ignores heading and sensor geometry. That is exactly the gap addressed here.

Two lines of work constrain the design. HeAL3D adds range and point-density heuristics to detector uncertainty. It uses those heuristics only as scalar multipliers, and it needs a detector that is already trained. Evidential interatomic potentials predict per-axis uncertainty but drive active learning from a learned head that is uncalibrated at step zero. Both confirm the lesson that Liu et al. describe as optimism bias: anything read off an untrained predictor should be kept out of the first round.

The method borrows its structure from FeatProp, which selects nodes using fixed, parameter-free propagated features instead of a trained network. Here the fixed quantity is sensor physics.

Building the system has three parts. The first computes precision tiles for every nuScenes keyframe from calibrated ego and sensor poses, with separate noise models for LiDAR, the 5 radars, and the 6 cameras. The second runs lazy-greedy maximization of $F$ under frame budgets. The third trains standard detectors on the selected frames. There is one secondary knob: a sensor-specific version, which scores only the modalities a given detector uses, against a pooled version. The noise parameters come from sensor datasheets and are fixed before any experiment. They are never tuned on detection accuracy.

Baselines

The naive reference is uniform random frame selection at the same budget, averaged over at least five seeds. PT4AL reports that on CIFAR-10 random selection reached 46.02% versus 55.83% for a good cold-start rule at iteration 1. So the gap on driving data is expected to be measurable but not huge.

The strongest existing competitor on this question is the Spatial+Temporal diversity rule of Exploring Diversity-based Active Learning. It uses Dijkstra distance on a GPS k-nearest-neighbor graph plus timestamp spread, and it is the only method in the file designed for exactly this setting: cold-start frame selection on nuScenes. It will be reimplemented and run on the same frame pool, at the same budgets of 600, 1200, 2400 and 4800 frames that paper uses, with the same detectors. It gets the same hyperparameter search budget for its neighbor count and term weights. Its feature-diversity term requires a partially trained backbone, so only its predictor-free terms apply at step zero. No reported result was found on Lacuna for its nuScenes mAP or NDS; the file only says it beat uncertainty sampling. Its numbers will therefore come from the reimplementation.

A second strong competitor is FreeSel. It passes camera images once through an off-the-shelf DINO vision transformer and applies k-means++-style diversity sampling over foreground patch features. It is the strongest predictor-free method that uses image content, and it will run on the front camera images with its published settings.

The fully supervised ceilings anchor the scale. nuScenes reports a top LiDAR baseline of 52.8% mAP and 63.3% NDS. RCM-Fusion reports 50.6% mAP and 58.7% NDS for radar-camera fusion.

Two ablations isolate the key ingredient. The isotropic ablation keeps the whole pipeline but sets $\sigma_r = \sigma_t$, so each cell's precision is round. This version still knows heading and field of view but loses directional noise. The pose-only ablation replaces $F$ with Euclidean coverage in (x, y, heading) space.

The method counts as beating the field if it exceeds the Spatial+Temporal rule and both ablations in mAP at the two smallest budgets across seeds, with non-overlapping 95% intervals, for the radar-camera detector.

Why not just…?

Why not just add heading to the GPS distance? This is the most serious threat. Much of the cold-start gain may come simply from not treating opposite-facing frames as duplicates. The pose-only ablation tests this directly. If it matches the full method, the honest conclusion is that orientation matters and noise anisotropy does not, and the proposal would shrink to that simpler finding. The distinguishing prediction is that anisotropy-aware selection lowers translation error (ATE) most for far-range objects seen by radar and camera, where transverse and depth blur dominate. A heading-aware distance has no mechanism to produce that pattern.

Why not just whiten the features and run a Coreset? A single global Mahalanobis metric is equivalent to Euclidean distance after a linear change of coordinates. It cannot represent uncertainty whose direction rotates from cell to cell with the line of sight. The anisotropy that matters here varies with each sample's sensor geometry. GCL-Coreset shows that the choice of metric space changes cold-start outcomes: 52.3 Dice versus 45.2 for standard Coreset at a 2% budget on ACDC. But a learned global metric still carries no ray geometry. A whitened FreeSel-feature Coreset will be included as an extra baseline to show this.

Why not just use HeAL3D after a small random warm start? At the budgets that matter, spending part of the budget on random frames and then trusting detector uncertainty is exactly the regime where optimism bias hurts. HeAL3D's reported gains, about 3% mAP over KECOR and CRB and full-supervision parity at 24% of labels on KITTI, come from well past the cold start. The two approaches are complementary rather than competing. The experiment below uses the proposed rule for round 0 and HeAL3D afterwards, and compares that against a random round 0 followed by HeAL3D.

Proposed Experiments

On nuScenes, selections are made from the training keyframes at the four budgets above. Three detectors are trained: a LiDAR-only VoxelNet, BEVFusion, and RCM-Fusion as the radar-camera model. Each is evaluated on the official validation split with mAP, NDS, and the per-error terms ATE and AOE, stratified by object range bins and by nighttime scenes. The nighttime split matters because nuScenes reports camera mAP falling 55–58% at night against 36% for LiDAR.

The first decisive check is the ordering of gains. The margin over the isotropic ablation should be largest for RCM-Fusion, intermediate for BEVFusion, and smallest for VoxelNet. If the ordering fails, anisotropy is not the operative cause.

The second check is a dose-response test. The ratio $\sigma_t/\sigma_r$ in the noise model is swept from 1 to its physical value. Detection gains should increase monotonically, and the chosen sets should show a measurable rise in crossing-view pairs, meaning cell observations from rays more than 60° apart.

The third check is round 0 followed by HeAL3D over three further rounds, to test whether a better seed set carries through to later active learning or washes out.

The fourth check is sensitivity: noise parameters perturbed by ±50% should not reverse the ranking against the Spatial+Temporal baseline. If they do, the method depends on calibration precision that practitioners will not have.