Few-Point Range Anchoring for Keypoint-Based Object Pose

A single camera cannot tell a small car nearby from a large car far away. Both project to the same pixels, so any camera-only estimate of an object's distance is a learned guess. Keypoint pipelines are a standard way to locate objects in 3D. They detect a few 2D landmarks, such as the projected corners of an object's 3D box, then use a geometric solver to recover the object's position and orientation. These pipelines inherit the same weakness: their distance is only as good as the predicted object size and the pixel accuracy of the landmarks, and both degrade with range. Cheap ranging sensors, such as a 4-beam LiDAR that returns fewer than 100 points per image, give exact metric distances. At range, though, they hit any one object only once or a handful of times.

This note proposes putting those few laser points directly into the object's pose solver. Each LiDAR point inside an object's image mask becomes one more residual: the 3D point should lie on the surface of the posed object model, alongside the usual requirement that model landmarks reproject onto the detected keypoints. One range point on a surface of known shape fixes the object's scale along the viewing ray once the keypoints have fixed its orientation. Two or three points also constrain yaw. The claim is that this coupling recovers metric object pose from one to five LiDAR hits. It should beat both camera-only keypoint detection and the obvious "use the median LiDAR depth in the box" shortcut specifically in that few-hit regime, and it should report when the camera and the laser disagree.

Core Claim and How It Differs From Prior Work

Prior work splits along the line this proposal crosses. On the keypoint side, Monocular 3D Object Detection via Geometric Reasoning on Keypoints predicts 14 car keypoints and recovers depth from height ratios against CAD templates. SMOKE predicts the projected 3D center as a keypoint and regresses depth from appearance. Real-time 3D Traffic Cone Detection for Autonomous Driving runs a Perspective-n-Point solver (PnP, which finds the rigid pose aligning known 3D model points to detected 2D points) against a cone model. Its error grows from 0.5 m at 10 m to about 1 m at 16 m. These three papers motivate the proposal: all are purely monocular, and the cone result shows the range-dependent failure directly.

On the sparse-sensor side, LiDARTouch and FusionDepth show that 4-beam LiDAR can fix the metric scale of dense depth maps. LiDARTouch reports 2.504 m RMSE against 2.112 m for full 64-beam supervision. However, both use the points for pixel-wise depth or camera ego-motion, never as constraints on an individual object's pose. FusionDepth converts its depth into a point cloud for a separate detector. LPCG uses LiDAR only to make training labels, so inference is camera-only.

On the RGB-D side, PCKRF and MK-Pose fuse keypoints with depth. Both assume dense depth over the object and, for PCKRF, a good initial pose. That assumption is exactly what fails at 4 beams.

The method comes from two sources. Hybrid Camera Pose Estimation builds minimal solvers and a sampling scheme over mixed correspondence types for camera localization in static maps. Fixing the Scale and Shift in Monocular Depth solves jointly for pose and unknown depth scale. We reuse the idea of mixing correspondence types, but apply it to a single dynamic object with a parametric shape rather than to a camera in a pre-built map.

Research Program

The object model is the KITTI-style oriented 3D box with predicted dimensions $\mathbf{d}$. The keypoints are its eight projected corners plus its center, in the spirit of the virtual control points of 3D Pose Estimation and 3D Model Retrieval for Objects in the Wild. This choice needs no extra keypoint annotation, because the corners come from standard box labels.

A detector in the style of SMOKE predicts the keypoints, their per-point uncertainty, the dimensions, and an instance mask. The LiDAR points $\mathbf{p}_j$ that project into the mask are treated as candidate surface hits. The pose is then

$$\hat{\mathbf{R}},\hat{\mathbf{t}}=\arg\min_{\mathbf{R},\mathbf{t}}\sum_{i}\frac{\|\mathbf{x}_i-\pi(\mathbf{K}(\mathbf{R}\mathbf{X}_i(\mathbf{d})+\mathbf{t}))\|^2}{\sigma_i^2}+\lambda\sum_{j}\rho\!\left(\frac{D_{\mathbf{d}}(\mathbf{R}^\top(\mathbf{p}_j-\mathbf{t}))}{\sigma_r}\right)$$

The symbols are defined as follows. $\mathbf{x}_i$ is a detected keypoint with predicted pixel uncertainty $\sigma_i$. $\mathbf{X}_i(\mathbf{d})$ is the matching box corner. $\mathbf{K}$ is the camera intrinsics matrix and $\pi$ is perspective division. $D_{\mathbf{d}}$ is the signed distance from a point to the surface of the box with dimensions $\mathbf{d}$. $\sigma_r$ is the LiDAR range noise. $\rho$ is a robust loss that suppresses background points leaking through the mask. $\lambda$ balances the two terms.

Robust initialization samples hybrid minimal sets: three keypoints plus one range point, or two keypoints plus two range points. Each set is scored by inlier count, following the adaptive solver selection in Hybrid Camera Pose Estimation. The solver is unrolled differentiably so that keypoint uncertainties are trained end-to-end through the final pose error.

A secondary output comes from the normalized residual of the range term at the optimum. It is a per-object consistency score: a large value means the image evidence and the laser disagree. We test whether this score flags the camera-only failure modes described by Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models. We treat that test as a diagnostic, not as the headline.

Baselines

The naive reference is camera-only SMOKE trained on the same KITTI split. It shows how much any LiDAR use helps. Alongside it we report the same detector feeding our solver with the range term removed ($\lambda=0$, pure keypoint PnP). That second configuration is the key ablation: it isolates whether the range residuals, rather than the PnP formulation, produce the gain. For orientation, Barabanau et al. report 50.82 AP3D at IoU 0.5 on KITTI Easy validation for the keypoint-geometry family. For SMOKE, no reported KITTI AP number was found on Lacuna. We will rerun it ourselves.

The most dangerous simple baseline is SMOKE plus a frustum-median correction. It takes the median depth of LiDAR points inside the predicted mask, adds half the predicted object depth along the ray to convert surface to center, and replaces SMOKE's depth with the result. Orientation and dimensions are left unchanged. It uses exactly the same points and costs nothing.

The strongest existing method is the FusionDepth route: dense depth completion from 4-beam LiDAR, conversion to a pseudo point cloud, and a standard detector. FusionDepth reports more than a 68% improvement over Pseudo-LiDAR++ on the KITTI leaderboard. No absolute AP3D value was found on Lacuna, so we will retrain it on our split. We also include LPCG as the "LiDAR at training time only" competitor, and SMOKE with a sparse-depth image channel as the learned-fusion competitor.

Fairness rules apply to all methods. Every method sees the identical simulated 4-beam downsampling of KITTI's 64-beam scans, using the protocol of LiDARTouch and FusionDepth. All share the same backbone (DLA-34), the same training schedule, and the same hyperparameter search budget. Beating the field means a statistically reliable AP3D gain over the frustum-median baseline in the 1–5-hit bins, with no loss in the 0-hit bin.

Why not just…?

Why not just replace the predicted depth with the median LiDAR depth in the box? This is the real threat. When an object carries dozens of hits, we expect it to match our method, and we will say so. It has three weaknesses. With one or two hits the median is a single point whose position on the car is unknown, so the surface-to-center offset is wrong by up to half the vehicle length. It cannot correct orientation. Background points inside a loose mask move it directly. The experiment stratifies objects by hit count, and if the median baseline is within one AP point of ours in every bin, the proposal's claim is refuted.

Why not just densify the LiDAR first and run any detector, as FusionDepth does? Densification spreads one laser hit into a disc of guessed depths and then discards the knowledge that those depths came from one physical point. A depth-completion error at an object boundary becomes an error the detector cannot see. Our solver keeps the raw point and weighs it against the keypoints explicitly. This path is also far heavier, since it adds a full depth network and a point-cloud detector. The hit-count stratification and a runtime comparison settle the question.

Why not just feed the sparse points as an extra input channel and let the network learn? That is the learned-fusion baseline above. With under 100 points per image, the extra channel is almost entirely zeros. Nothing forces the network to respect geometry on the rare object a point lands on, and the CRAB analysis of depth ambiguity in learned camera-radar fusion (CRAB) suggests that learned lifting still leaves ray-direction ambiguity. If this baseline matches ours, the explicit solver is unnecessary and we would report that.

Proposed Experiments

The main experiment is on KITTI 3D detection, cars and pedestrians, with 4-beam simulation. We report AP3D and BEV AP at standard IoU thresholds, plus mean depth error and yaw error. Every metric is broken down by the number of LiDAR hits per object (0, 1, 2–5, 6–20, more than 20) and by distance band.

The decisive check is the 1–5-hit, more-than-20-m slice, which is where the claim lives. A secondary check repeats the study on nuScenes. That dataset's sparser native LiDAR and 360-degree cameras test transfer without simulation, and it lets us swap LiDAR for radar points with an inflated $\sigma_r$. We expect the gain to shrink with radar's poor angular resolution, and measuring how far it shrinks is itself informative.

A third check perturbs the camera-LiDAR calibration by up to 1° and 10 cm. Misalignment is a known fusion failure. The robust loss should degrade gracefully, while the median baseline should shift systematically.

Finally, we evaluate the consistency score as a detector of wrong poses, using precision–recall for flagging objects whose depth error exceeds 2 m. We also test it on synthetic flat road markings rendered into KITTI images, which should yield strong keypoints but laser points lying on the road plane.