LiDAR-Checked Decoding: Verifying Vision-Language Claims Against Range Geometry in Constant Time

Vision-language models are now being asked questions about driving scenes, such as whether there is a barrier ahead or how many trucks are parked on the right. They often answer fluently and wrongly. The usual fixes for this "hallucination" all check the answer against the same camera image that misled the model. That cannot help when the image itself is ambiguous. A camera-only driving model will read a flat black line painted on asphalt as a raised curb and steer off the road (Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models). A decoder that amplifies "what the image says" will amplify that mistake too. A car carrying LiDAR, a laser scanner that returns 3D points, already holds an independent witness. A painted line has no height and a curb does. We propose to make that witness speak at the moment the language model commits to a claim.

The core idea is simple to state. Whenever the model is about to emit a physically checkable claim, such as "yes, there is a pedestrian left of the bus" or "three cars", we look up the matching 3D region in a precomputed summed-area table of LiDAR occupancy and height. A summed-area table, or integral image, is a cumulative-sum grid that returns the total over any box with a handful of memory reads. We then nudge the model's token probabilities toward the answer the geometry supports. An external knowledge source supplies the expected physical footprint of each claimed object class, for example a bus being about 3 m tall and 12 m long. This tells the check what geometry to look for, including for rare classes the model's detector was never trained on. The integral table is what makes the check affordable at the query rates these benchmarks imply. DriveLM-nuScenes averages 91.4 question-answer pairs per frame (DriveLM), so the checks must cost almost nothing.

Core Claim

Sensor-geometric evidence injected at decision tokens reduces hallucination in driving vision-language models on two kinds of question that camera-side decoding fixes cannot reach. The first is existence questions with adversarial co-occurring negatives. The second is "phantom geometry" cases where the image itself is deceptive. The check should run at a small fraction of the cost of contrastive decoding, because it adds no extra language-model forward pass. We further claim it beats the most obvious competitor, which serializes a 3D detector's boxes into the prompt. That competitor inherits the detector's misses, and NuScenes-QA accuracy drops from 84.3% with ground-truth boxes to 60.4% with detected boxes (NuScenes-QA). Our check reads raw geometry and never needs a detection.

How the Pieces Fit, and Prior Work

Multimodal hallucination is the problem we target. POPE showed that yes/no existence probing is a stable way to measure it. It also showed that adversarial negatives, meaning absent objects that commonly co-occur with present ones, drive errors up, and that some models answer "yes" up to 99% of the time. Three mitigation lines work entirely inside the 2D camera channel. M3ID contrasts image-conditioned with text-only token probabilities and needs a second forward pass. RITUAL decodes over randomly transformed copies of the image. SHARP steers middle-layer hidden states. All three assume the camera features are the ground truth to be amplified. We keep their decision-time framing but replace the source of evidence. Villa et al. found that models sometimes answer correctly from scene priors while failing to see the object. That motivates evaluating with probes whose priors and geometry disagree.

Sensor fusion supplies the evidence and its limits. NuScenes-QA fuses camera and LiDAR in a bird's-eye-view grid, a top-down map of the scene around the car, and answers questions with a fixed reader. It never asks whether wrong answers are prior-driven hallucinations, and its counting accuracy is only about 23%. STSBench shows the gap we are exploiting. GPT-4o given ground-truth 3D coordinates as text scores 57.08%, above camera-based driving models (DriveMM 39.51%, OmniDrive 29.33%) and a general model (InternVL 2.5 8B, 46.07%).

Two sensor papers constrain the design. Gao shows LiDAR cannot tell tall grass from a solid obstacle at range. The check may therefore only arbitrate geometric content, such as presence, extent and elevation, and must leave material and semantic identity to the camera. Heidecker et al. describe corner cases in which both sensors degrade together, as with spray producing spurious returns. The check must therefore abstain when LiDAR point density in the region is unreliable.

Integral images supply the mechanism. The four-lookup box sum dates from Viola-Jones. Sanin et al. extend it to 3D volumes, which we reuse to build a cuboid table over (x, y, height-bin). Local Context Normalization keeps tables of both values and squared values to get local mean and variance in constant time. We apply the same trick to LiDAR height, so the variance of height inside a box separates a flat road marking from a raised curb. Scharr et al. project 3D volumes into 2D integral silhouettes, a 102× CPU speedup. That is the closest precedent for using integral tables as a fast geometric consistency check, but it has no ranging sensor and no language. Box convolutions warn that GPU gains over optimized dense kernels are modest. We take that seriously below.

External knowledge plays a deliberately narrow role. EnFoRe and INFOSEEK show that retrieval only helps once the entity is correctly identified. INFOSEEK reports 18.2% accuracy for a CLIP+FiD pipeline against 45.6% with an oracle entity linker. In our setting, the language model's own claim supplies the entity, and the knowledge source is queried for that entity's physical signature: height range, footprint, and whether it is elevated or flat. That is the sense in which knowledge integration enters. It converts "a street sweeper is there" into a geometric hypothesis the sensor can test. Distractor retrieval is a known failure here. The ablations therefore compare knowledge-sourced signatures with signatures learned from training data.

Research Program

For each frame we build a 3D integral table $S(x,y,z)$, where each cell holds the cumulative count of above-ground LiDAR returns in bird's-eye-view cell $(x,y)$ and height bin $z$. We also build 2D tables of summed height and summed squared height. Construction is one prefix-sum pass over the grid. Any axis-aligned cuboid $R$ then yields its point count $N_R$ with eight lookups. Its mean height $\mu_R$ and height variance $\sigma^2_R$ come from

$$\mu_R = \frac{H_R}{N_R}, \qquad \sigma^2_R = \frac{Q_R}{N_R} - \mu_R^2,$$

where $H_R$ and $Q_R$ are the box sums of height and squared height. The grid is axis-aligned in the ego frame, so rotated objects and camera-ray frustums are covered with a small union of axis-aligned strips, the standard workaround for this known limitation.

A claim parser maps the decision context to a region and a class. The context can be a question's referenced object and spatial phrase, as in NuScenes-QA templates, or a DriveLM perception answer containing pixel coordinates, which we lift to a ray frustum in bird's-eye view. The knowledge lookup returns a height interval and footprint for the class, which fixes the cuboid's vertical extent and the expected fill.

From these statistics a small calibrated model produces an evidence score $s_R$. This is a log-likelihood ratio between "class-sized object present" and "absent", fit on the training split. At the decision token we add it to the answer logits, $z'_a = z_a + \lambda\, s_R\, \mathrm{sign}(a)$. Here $a$ ranges over the answer options, $\mathrm{sign}(a)$ is $+1$ for affirming answers and $-1$ for denying ones, and $\lambda$ is a single weight tuned on validation. When $N_R$ falls below a range-dependent density floor, $s_R$ is set to zero and the model answers alone. For counting, the occupied above-ground area in the region divided by the knowledge-sourced footprint gives a prior over counts. That prior reweights the count tokens.

Nothing in the language model is retrained. This keeps the comparison with decoding-time methods clean.

Baselines

The naive reference is the unmodified vision-language model with greedy decoding on camera input. On STSBench this is InternVL 2.5 8B, reported at 46.07%. On NuScenes-QA and DriveLM-nuScenes we run the same model ourselves, since the file has no reported number for it there.

The strongest existing camera-side fixes are M3ID, RITUAL and SHARP, all run on the same base model. For SHARP the published reference points are 82.8% adversarial POPE accuracy on LLaVA-Next against a 79.5% baseline, and 7.04 s per sample against 13.97 s for visual contrastive decoding. For RITUAL the reference is 88.87% POPE accuracy on LLaVA-1.5 against 84.13%. These are 2D benchmarks, so we re-run every method on our driving probes using the authors' default hyperparameters and an equal validation-tuning budget, matching the one weight $\lambda$ we tune. Beating them means higher adversarial existence accuracy at lower latency, measured on the same GPU.

The strongest competitor on this specific question is detection serialization. Here we run the same LiDAR-camera 3D detector used by the NuScenes-QA baseline and write its boxes into the prompt as text, mirroring STSBench's coordinate-text setting. Its ceiling is visible in the reported 57.08% for GPT-4o with ground-truth coordinates on STSBench and 84.3% versus 60.4% for ground-truth versus detected boxes on NuScenes-QA. We also report the NuScenes-QA fusion model itself at its reported 60.4%. Beating detection serialization means higher accuracy with detected boxes on existence and counting, and especially on long-tail or non-detector classes, at equal or lower token budget.

Three ablations isolate the ingredients. A brute-force point-in-cuboid check computes identical statistics; it must match our accuracy exactly and differ only in latency, which shows what the integral table contributes. Learned class signatures replace knowledge-sourced ones, which isolates the knowledge source. Depth-completed camera pseudo-points in the style of SFD replace LiDAR, which tests whether an active sensor is needed or monocular depth suffices.

Why not just…?

Why not just serialize the detector's boxes into the prompt? This is the real threat. If the detector covers the asked-about classes and the scene is small, it may match us. Our bet rests on two documented weaknesses. Detection errors cost 24 points on NuScenes-QA. Long serialized contexts are exactly where conditioning dilution, documented by M3ID, erodes attention to evidence. The experiment that settles it is the per-class breakdown on NuScenes-QA. If serialization ties us on the classes the detector covers and loses only on classes it does not cover, we will report the method as a complement for open-vocabulary claims rather than a replacement.

Why not just use contrastive or augmentation decoding like M3ID or RITUAL? These methods compare the model against itself on camera input. When the camera is fooled, as with painted lines, the conditioned and unconditioned passes agree on the phantom, so contrast has nothing to subtract. The phantom-geometry probes described below test this directly. A camera-side method that fixes phantoms would falsify our motivation.

Why not skip the integral table and count points in each box on the GPU? For a few queries per frame that is likely just as fast, and the box-convolution results already warn that GPU gains are modest. The table pays off when many queries share one frame and when queries cover many scales and height bands. We will plot latency against queries per frame, from 1 up to DriveLM's density, for both implementations. If the curves never separate, we will say the integral table is an implementation convenience and not a contribution.

Why not use a bigger model that has absorbed the geometry from data? Scale does not remove prior-driven yes-bias in POPE. INFOSEEK's closed-book PaLI-X reaches only 10.8%. A larger base model is still run as a reference, to check that our gains persist instead of shrinking to nothing.

Proposed Experiments

The first experiment builds a driving version of POPE on nuScenes. We sample existence queries in Random, Popular and Adversarial settings from 3D annotations, with adversarial negatives drawn from classes that co-occur in nuScenes scenes. We report accuracy, F1 and yes-rate.

The second targets phantom geometry. We replay the painted-line attack in CARLA with simulated LiDAR at the 90–115° angles that the original paper found most damaging. We ask "Is there a barrier or curb ahead?" and measure false-yes rate, plus steering deviation when answers feed a simple planner.

The third covers the standard benchmarks. We evaluate NuScenes-QA by question type, with counting as the hardest at roughly 23%. We evaluate STSBench by interaction level, expecting little gain on agent-to-agent motion questions that geometry snapshots cannot answer; this is an honest negative control. We evaluate the perception stage of DriveLM-nuScenes and track whether fewer perception hallucinations reduce errors downstream in the planning graph.

Latency is reported as milliseconds per query and per frame against the brute-force check and against the extra forward passes of contrastive decoding.

Two results decide the proposal. First, on adversarial existence and phantom-geometry probes, LiDAR-checked decoding must beat both M3ID-style decoding and detection serialization. Second, the camera pseudo-point ablation must fall clearly short of real LiDAR. If the second fails, the active sensor is not the reason the method works, and the contribution reduces to depth-aware decoding.