Calibration-Aware Object Tokens for Driving Vision-Language Models

When a car's camera and LiDAR (a laser scanner that returns 3D points) drift out of calibration, a 3D detector usually just loses accuracy. A vision-language model fed the same fused features may fail in a worse way. It describes a pedestrian "on the left, crossing" because camera pixels from the wrong place were attached to a correctly placed LiDAR object, and its language prior fills in a plausible story. This note proposes to study that failure directly and to fix it with one specific mechanism. Each object the model reasons about is built by soft, learnable camera-to-LiDAR matching, and each carries an explicit score for how well the two sensors agree about it. The language model then sees both the object and how trustworthy its fused evidence is, so it can hedge or decline instead of hallucinating.

The opportunity follows from a documented gap. On a driving benchmark, a text-only GPT-4o given ground-truth object trajectories scores 57.08%, while vision-language models reading raw camera video score between 29.33% and 46.07% (STSBench). Perception, not reasoning, is the bottleneck. Sensor-fusion research has already found ways to keep cross-sensor features usable under miscalibration. Hallucination research has identified when language models override weak visual evidence. Neither field has asked what happens to generated language when the fusion itself is wrong, or whether the fusion module can report that it is wrong.

Core Claim

Our claim is that physical sensor misalignment is a distinct and measurable driver of spatial hallucination in multi-sensor vision-language models. The fix has two parts. Soft cross-modal matching keeps object features correct under moderate drift. A per-object agreement signal lets the language model stop trusting evidence once drift exceeds what matching can repair. The novel and testable part is the second one.

Prior work supplies the matching. AutoAlign lets 3D voxel queries cross-attend over the whole image instead of trusting a projection matrix. TransFusion aggregates camera features into sparse 3D box candidates with an attention decoder and reports robustness to miscalibration. CF-Deformable DETR uses deformable attention, which samples a few learned offsets around a projected point, to find shifted infrared-visible correspondences. All three end in box-regression heads and never produce language. None of them exposes how far the model had to search, or how much the two sensors disagreed, as a quantity that downstream reasoning could use.

Research Program

The system has three stages. The first is a LiDAR-camera fusion encoder in the TransFusion style. Its object queries are anchored in 3D from LiDAR and gather camera features through deformable cross-attention around each query's projected image location.

The second stage computes an agreement score $a_i \in [0,1]$ for each object query $i$. A small head reads three signals: the magnitude of the learned sampling offsets, the entropy of the cross-attention weights, and the feature consistency between the LiDAR-side and camera-side embeddings, in the spirit of CF-Deformable DETR's point-level consistency loss. The head is trained with free supervision. During training we randomly perturb the camera-LiDAR extrinsics, the rotation and translation between sensors, by known amounts. The head learns to predict whether the residual misalignment at that object exceeds the error that offset search can recover. Because we inject the perturbation ourselves, we always have the label.

The third stage feeds each object to the language model as a short group of tokens: the fused feature, its 3D box, and an embedding of $a_i$. This places the uncertainty where the model reads its evidence. That matters because hallucination work shows that visual evidence weakens with depth and with response length (Zou; Why LVLMs Are More Prone to Hallucinations in Longer Responses). A scalar applied only at the end would come too late.

Training data are question-answer pairs generated from nuScenes training annotations. They cover ego motion, agent motion, and agent-to-agent relations. Some targets explicitly say "cannot determine" when the injected perturbation at the queried agent is unrecoverable. STSBench's 43 scenario types are built on the nuScenes validation split, so we hold out whole scenario families from training to avoid template leakage.

The analysis half of the program matters as much as the method. We sweep extrinsic rotation and translation error, plus camera-LiDAR time offset, from zero to well beyond the drift deformable attention tolerates. That beyond-tolerance regime is the "extreme displacement" limit the file notes for deformable methods. At each level we measure how answer accuracy, hallucination rate, and the model's stated confidence co-vary.

The prediction that would establish misalignment as a separate cause is specific. Rigid-projection fusion should show hallucination rising while confidence stays flat, which is the signature of the language prior taking over that POPE and MERLIM document for single images. MERLIM's version is models still asserting objects after inpainting removed them. Our model should instead show hallucination held down while abstention rises.

Baselines

The naive reference point is a camera-only generalist vision-language model on STSBench. The strongest such number in the file is InternVL 2.5 8B at 46.07% accuracy (STSBench). We rerun it on our held-out split, and on our hallucination probes with no perturbation, so that the comparison uses identical questions. We also fine-tune its connector on our generated QA data with the same step budget, which separates data effects from architecture effects.

The strongest competitor is a detect-then-describe pipeline. A LiDAR-camera detector produces boxes and tracks, these are serialized as text, and a language model of the same size reasons over them. This is the most dangerous baseline because STSBench's 57.08% oracle shows that good coordinates as text already beat every vision-language model. A strong detector such as AutoAlign reports 66.6 mAP and 71.1 NDS on nuScenes (AutoAlign), so this pipeline might close much of the gap.

We run it with the same fusion encoder as our model, so detector quality is matched, and the same language backbone and fine-tuning data. A second variant also serializes the detector's box confidence, which gives text-side uncertainty a fair chance. Beating this baseline means one of two outcomes. Either our model is equal at zero perturbation and has a clearly lower spatial-hallucination rate at moderate and severe perturbation, or it has higher accuracy among answered questions at matched coverage. If the pipeline with serialized confidence matches us on both, the agreement token adds nothing new, and we would report that.

The rigid-projection baseline is identical to our model except that camera features are sampled exactly at the projected point, with no learned offsets. It isolates the contribution of soft matching.

The key ablation is our full model with the agreement embedding removed, trained identically. If removing $a_i$ leaves the hallucination-versus-perturbation curve unchanged, the central claim fails. A second ablation replaces the learned head with raw offset magnitude alone, to test whether the trained agreement predictor matters.

All variants share the language backbone, the generated training set, the optimizer budget, and three seeds per configuration.

Why not just…?

Why not just serialize detector boxes into the prompt and let the language model reason? This is the real threat, and the Baselines section is built to settle it. The STSBench oracle suggests coordinates-as-text is powerful. The open question is what happens when the coordinates are wrong without warning. Under miscalibration a detector still outputs crisp boxes, and text serialization drops appearance and any notion of cross-sensor disagreement, so the language model has no reason to doubt them. If a confidence-annotated text pipeline matches our curves, we will conclude that the gain comes from the detector, not from object tokens. We would then reframe the contribution around the self-supervised agreement head, which can feed either interface.

Why not just prompt for a spatial scratchpad, emitting coordinates before answering? The coordinate-anchored reasoning line (Coordinate Anchored Reasoning for Multimodal Models) shows that intermediate coordinates help when perception is sound. The coordinates are produced by the same misaligned features, though, so under drift they are themselves hallucinated. We test this directly by running scratchpad prompting on the naive and rigid baselines across the perturbation sweep. We also score the scratchpad coordinates against ground truth, which shows whether errors start in perception or in reasoning.

Why not just use contrastive decoding such as VCD or RITUAL? VCD subtracts logits computed on a noise-corrupted image and reports up to +7.4 POPE F1. RITUAL contrasts randomly flipped or cropped views and moves LLaVA-1.5 from 84.13% to 88.87% POPE accuracy. Both double decoding cost, and both target object-existence hallucinations driven by language priors, not sensor disagreement. RITUAL's image flips would also break camera-LiDAR geometry. We include a stronger, setting-adapted variant whose negative view is the same scene fused under a deliberately worsened calibration. If this cheap contrast matches the agreement token, the token is unnecessary.

Why not just augment a standard model with calibration jitter during fine-tuning? Augmentation may teach robustness to small drift, but it gives the model no way to detect large drift. We run it as an extra condition on the rigid-projection baseline. The expected signature is improved accuracy at small perturbations but the same flat-confidence hallucination at large ones.

Proposed Experiments

Evaluation uses three instruments. The first is STSBench multiple-choice accuracy (STSBench dataset) on held-out scenario types. We report the Agent-to-Agent category separately, where the benchmark already finds a 20–30% gap relative to simpler tasks.

The second is a driving analogue of POPE. We build existence and relative-position yes/no probes from nuScenes annotations with random, popular, and co-occurrence-adversarial negatives. We report the "yes" rate because Li et al. found it reaching nearly 99% for some models (POPE).

The third is a spatial analogue of CHAIR, the share of mentioned objects that are not actually present, adapted to claims about an agent's position, distance band, and motion state in free-form scene descriptions. Every claim is checked by rule against nuScenes boxes, not by a vision-language judge, because automated judges themselves hallucinate scene layouts (Chen et al.). This applies to every system, including any judge-free scoring of baselines.

The decisive check is the perturbation sweep, plotted as hallucination rate and selective accuracy against calibration error for every system. The proposal succeeds if three things hold at moderate-to-severe drift. Our model's spatial-hallucination rate stays well below that of the rigid and text-pipeline baselines. Its abstentions concentrate on the objects that were actually unrecoverable, which we can verify because the injected error is known per object. Removing the agreement embedding erases the advantage.

A secondary check concerns layout. Concentric reordering of visual tokens (CCA) assumes a central focus, while MCA-LLaVA argues for treating space symmetrically. We therefore break down hallucination by azimuth around the vehicle. This confirms that object-token inputs do not create blind spots at the periphery, where approaching agents appear in a 360-degree camera rig.