Return-Guided Pooling for Anchor-Free 3D Detection with Cheap Sensors

A low-cost car perception stack pairs a stereo camera with a four-beam LiDAR (about $600, roughly 5% of the price of a 64-beam unit). It can estimate depth fairly well after correction, yet it still misses and misplaces distant cars and pedestrians. We argue that much of this remaining error does not come from the depth estimate. It is created inside the detector. Image-based 3D detectors now often feed per-pixel metric coordinates $(x, y, z)$ into an ordinary 2D convolutional network. Such a network shrinks its feature maps by averaging neighboring pixels. For appearance features, averaging only blurs. For coordinate channels, averaging a distant car's pixels with the wall behind it produces a 3D point halfway between them, in empty space where nothing exists. A center-point detector reads the whole box (size, depth, heading) from the single feature vector at the object's center. When that center cell holds invented geometry, the box is wrong even if the input depth was right.

The cheap LiDAR is the natural fix. Its four beams are too sparse to detect objects directly, but each return is an exact metric measurement of a surface. We propose to use these returns to steer how the network downsamples. Pixels whose depth agrees with a nearby return, meaning they lie on a real surface, are kept together. Pixels at other depths are not averaged into them. The object's center cell therefore survives the trip to low resolution. In this design the sparse sensor's job is to protect resolution, not to be a detection input.

Core Claim

In camera plus ultra-sparse LiDAR 3D detection, the long-range accuracy gap that remains after depth correction is dominated by feature-map downsampling that mixes foreground and background geometry at object centers. Replacing average and strided downsampling of coordinate features with pooling weighted by depth agreement with sparse LiDAR returns will close a substantial part of that gap for an anchor-free center-point head. The gain should come at little extra compute, and it should be concentrated at 40–70 m and on small classes.

The claim has a diagnostic half and a method half, and the diagnostic half can fail on its own. It is possible that once depth is corrected, the residual error is mostly depth noise rather than mixing. The first experiment is designed to find that out before any method is built.

Where This Sits

The cost-accuracy line of work established the sensor regime and the representation we build on. Pseudo-LiDAR++ builds stereo cost volumes in depth space to limit the quadratic growth of depth error with distance. It then uses Graph-based Depth Correction (GDC), which spreads the exact depth of 4-beam returns across image surfaces. It still reports a gap to 64-beam LiDAR at 50–70 m under IoU ≥ 0.7, and it pairs the corrected points only with anchor-based detectors. End-to-End Pseudo-LiDAR lets detection loss flow back into the depth network, but it downsamples uniformly. Rethinking Pseudo-LiDAR Representation (PatchNet) shows that the benefit of pseudo-LiDAR comes from the change to world coordinates, not from the point-cloud format. It feeds $(x, y, z)$ maps as image channels to a ResNet-18. We reuse this representation because it is cheap. We also identify its weak point: the ResNet's averaging operates directly on metric coordinates. Our work is the first to treat that averaging as the problem. These three papers motivate the regime and supply the representation.

The anchor-free line supplies the detection head and shows how fragile it is. CenterPoint finds objects as peaks in a center heatmap and regresses box attributes at the peak. It notes that its second-stage refinement breaks down when objects have few surface points. MonoPair applies the same center-point idea to monocular 3D detection with a DLA-34 backbone. Tiny lesion detection argues that center points avoid the anchor-overlap failure on tiny targets. We accept that argument, and it is why we keep an anchor-free head. Our point is that this advantage disappears if the center cell's feature has been corrupted before the head sees it.

The resolution-degradation line supplies the pooling mechanism. LIP replaces fixed pooling with importance weights learned by a small convolution. It raises Faster R-CNN small-object AP on COCO from 21.9 to 24.0. Its weights come from image content alone, and it has never been tested on metric coordinate maps or 3D heads. DPNet predicts a downsampling factor per image. It shows that brute-force upscaling wastes compute, reporting a 35.93% GFLOP reduction on TinyCOCO at equal or better mAP. GLAD runs a full-resolution local branch around tiny targets. These papers constrain the design: they tell us not to answer with brute-force resolution. We differ from them in the source of the pooling weights. We take them from an independent metric sensor rather than from learned image saliency.

On the fusion side, BEVFusion and DGFusion fuse dense 32-beam LiDAR with cameras in a bird's-eye grid. DGFusion explicitly flags distant, sparse objects as its hard cases. Both assume dense LiDAR, and both pool uniformly. Distillation from LiDAR shows that teaching a camera model from LiDAR helps localization but not semantics. This is why we keep the cheap LiDAR at test time rather than distilling it away.

Research Program

The pipeline begins with the Pseudo-LiDAR++ stereo depth network and GDC, run with 4-beam returns, which gives a corrected per-pixel depth map. That map is converted to $(x, y, z)$ channels and concatenated with RGB, following PatchNet. The result passes through a ResNet-18 or DLA-34 backbone to an anchor-free head. The head predicts a center heatmap in image space, plus sub-pixel offset, depth, 3D size and heading at each peak, in the style of MonoPair without its pair graph.

The single change is at every downsampling step, in both the coordinate channels and the feature channels computed from them. Take a pooling window $W$. Let $\hat z_W$ be the depth of the LiDAR return nearest the window, projected into the image and propagated along the GDC graph so that every window has one. Each pixel $i$ in the window with depth $z_i$ then receives a weight

$$w_i = \exp\!\left(-\frac{|z_i - \hat z_W|}{\tau\, \hat z_W}\right)\cdot \sigma(g_i),$$

where $\tau$ is a relative depth tolerance, scaled by $\hat z_W$ because stereo error grows with range. The term $\sigma(g_i)$ is a LIP-style learned importance gate. The pooled output is the $w$-weighted mean. Windows that contain several depth modes but no nearby return fall back to the learned gate alone. This makes the design honest about windows the four beams never reach.

The first deliverable is a measurement, not a model. For every ground-truth object, we will record how far the center-cell coordinate feature at each backbone stage drifts from the true object depth. We will then relate this "center contamination" to box error as a function of range.

Baselines

The naive reference is PatchNet-style coordinate maps from stereo alone with a standard ResNet-18 and center-point head. It shows how far stereo alone gets and anchors all relative claims.

The strongest existing competitor on this exact question is Pseudo-LiDAR++ with GDC on stereo plus simulated 4-beam LiDAR, run from its released code (mileyan/Pseudo_Lidar_V2). It is the only method in the file built for this sensor budget. The file reports only relative figures for it: 10% over prior stereo methods, 40% on faraway objects, and a further 27% relative gain from 4-beam LiDAR. No reported absolute AP result was found on Lacuna for the 4-beam configuration, so we will reproduce it ourselves. We will run it twice. The first run uses its native anchor-based detector. The second places its GDC-corrected depth into our exact anchor-free pipeline with uniform pooling. The second run is the fair comparison: same corrected depth, backbone, schedule, input resolution and hyperparameter search budget, with three seeds each.

Two ablations isolate the key ingredient. Plain LIP pooling, with learned weights and no return term, tests whether the sparse LiDAR adds anything beyond learned saliency. A non-learned depth-mode pooling, which keeps the dominant depth cluster in each window, tests whether learning adds anything beyond a heuristic.

For upper-bound context, Deep Continuous Fusion reports 82.54% AP on KITTI Easy 3D with full 64-beam LiDAR. MonoPair reports car $AP_{3D}$ of 10.42% at IoU 0.7 on validation with a monocular camera. These bracket the cheap-sensor regime. On nuScenes, BEVFusion reports 70.2% mAP with full 32-beam LiDAR. No reported result was found on Lacuna for any method on beam-subsampled nuScenes.

Beating the strongest baseline means a statistically consistent gain over GDC with uniform pooling in the same anchor-free pipeline, in KITTI Moderate and Hard car $AP_{3D}$ at IoU 0.7 in the 40–70 m bin, with smaller or zero gains under 20 m. It also means a gain over plain LIP, which is required for the claim that the cheap sensor's role is resolution protection.

Why not just…?

Why not just run at higher input resolution or a finer output stride? This is the most obvious fix, and we include it as a compute-matched arm. We run the uniform-pooling pipeline at 2× input and at stride 2 and report GFLOPs and latency next to AP. DPNet shows that brute-force upscaling inflates compute and background negatives for small-object detection. That evidence comes from 2D imagery, though. If 2× input matches our method at similar cost on KITTI, the claim reduces to "resolution matters," and we would report that.

Why not just use learned importance pooling like LIP without the LiDAR? LIP's weights are learned from image appearance. At 60 m, a car and the wall behind it can look alike in a few-pixel window while their depths differ by meters, and the four beams measure exactly that difference. This is a real threat. The LIP-only ablation settles it: if LIP matches return-guided pooling, the LiDAR term is unnecessary and we will say so.

Why not just rely on GDC depth correction, which already uses the 4 beams? GDC fixes per-pixel depth, but a corrected depth map is still averaged by the backbone. The contamination diagnostic tests this directly. If center contamination stays small after GDC, the premise is false and the project stops at the diagnostic. If contamination is large and predicts box error, GDC alone is insufficient by construction.

Why not just distill a LiDAR teacher into a camera-only model? The distillation study finds that distillation transfers localization but not attributes. nuScenes reports a 55–58% nighttime mAP drop for vision methods versus 36% for LiDAR. We therefore keep the $600 sensor at test time and do not add a distillation arm.

Proposed Experiments

The main benchmark is KITTI 3D. We use stereo pairs and 4-beam LiDAR simulated by subsampling the 64-beam scans, as in Pseudo-LiDAR++, and we acknowledge that this is not native 4-beam hardware. Metrics are $AP_{3D}$ and $AP_{BEV}$ at IoU 0.7 for cars and 0.5 for pedestrians and cyclists, reported by difficulty and by 0–20, 20–40 and 40–70 m range bins.

The first decisive check is an oracle study. We substitute ground-truth LiDAR-derived depth for stereo depth while keeping uniform pooling. If long-range AP remains far below the 64-beam reference with perfect input depth, downsampling is shown to cost accuracy independently of depth quality. We add a correlation, per object, between center contamination and box center error.

The second check is the main comparison against the baselines above, at matched FLOPs.

The third check sweeps the beam count from 1 to 16. Return-guided pooling should help most at intermediate sparsity. Its advantage should shrink at 16 beams, where uniform pooling on accurate depth suffices, and at 1 beam, where most windows fall back to the learned gate. This pattern would distinguish the mechanism from generic regularization.

For transfer, we train and test on nuScenes with front cameras and the 32-beam LiDAR subsampled to 4 beams. We report mAP and NDS with the class breakdown for traffic cones, pedestrians and bicycles, and repeat the nighttime split, where the LiDAR-derived weights should matter most.