Answering Driving-Scene Relations from the Sensor That Can Actually See Them

Vision-language models used for driving fail most often on a specific kind of question: what two other road users are doing to each other. Examples are whether a cyclist is cutting in front of a bus, or whether a pedestrian is yielding to a turning van. On the STSBench driving benchmark, a text-only language model given ground-truth 3D coordinates beats every model that reads camera pixels. Every model also loses 20–30 points on these agent-to-agent questions compared with questions about the ego vehicle. Our reading of that gap is that the models answer geometric questions from camera images and language habits, and so invent relations that the metric geometry would rule out. We propose to split each relation into two parts and send each part to the sensor that measures it. Motion relations such as "following," "cutting in," "crossing the path of" and "yielding to" are scored by a small model that sees only the two agents' metric 3D tracks from LiDAR. Appearance relations such as "carrying," "pushing" or "signalling" are scored by a vision-language model, but only as a likelihood over a fixed set of candidate answers, never as free text. A per-modality reliability check decides whether each part can be trusted for this pair of agents in this frame. If it cannot, the system abstains instead of guessing.

The bet has two halves. Because the motion scorer never sees what class the agents are, a relation learned mostly from car–car pairs should carry over to rare pairs such as cyclist–bus or truck–pedestrian. These rare pairs are where driving logs are thinnest. The reliability gate should then turn sensor failures into abstentions rather than confident inventions. Examples of such failures are camera glare, LiDAR sparsity in the far field, and a painted pattern that looks like an obstacle. The deliverable is a relation-answering module for driving VLMs, plus a compositional and degraded-sensor evaluation built on nuScenes. Together they test whether relational hallucination in driving is mainly a problem of sending each question to the wrong evidence.

Core Claim

Our claim is specific. A large share of relational hallucination in driving VLMs comes from motion predicates being answered without metric evidence. A predicate is the relation word in a ⟨subject, predicate, object⟩ triplet. If that is true, factoring triplets so that motion predicates are computed only from relative 3D tracks will do two things. It will close most of the pixel-versus-coordinates gap that STSBench documents, using perception tracks rather than ground truth. It will also generalize to subject–object combinations never seen in training, because the motion term cannot entangle with object appearance. Formally, for agents $a$ and $b$ and predicate $p$, we score

$$ s(a,p,b) = \log P_{\text{det}}(a) + \log P_{\text{det}}(b) + r_L\, \alpha_p\, g_p(\tau_a,\tau_b) + r_C\, \beta_p\, v_p(I_{ab}). $$

The terms are as follows. $P_{\text{det}}$ is the open-vocabulary 3D detector's class confidence. $g_p$ is the motion scorer applied to the two agents' metric tracks $\tau_a, \tau_b$ over a past-and-forecast window. $v_p$ is the vision-language model's candidate-token likelihood for $p$ given multi-view crops $I_{ab}$ of the pair. $\alpha_p, \beta_p \ge 0$ are learned per-predicate weights saying how much each predicate depends on geometry versus appearance. $r_L, r_C \in [0,1]$ are reliability estimates for LiDAR and camera on this pair. The system abstains whenever the dominant evidence term for the top-scoring predicate has reliability below a threshold.

How This Differs from Existing Work

The prior work falls into three roles: some papers motivate the proposal, some supply its methods, and some are its nearest competitors. STSBench motivates it. That benchmark mines 43 interaction scenarios from nuScenes into 971 multiple-choice questions with hard negatives. It found text-only GPT-4o with ground-truth coordinates at 57.08% against InternVL 2.5 8B at 46.07%, DriveMM at 39.51% and OmniDrive at 29.33%. STSBench diagnoses the problem but proposes no remedy, and its strongest result uses oracle coordinates.

DA-HOI supplies the appearance half of our method. It decouples localization from interaction recognition by feeding open-vocabulary detector boxes to a multimodal LLM. It then scores candidate verbs in one forward pass instead of generating text. DA-HOI works only on static 2D images and folds 2D box geometry into the language model. We take the geometry out of the language model entirely and make it metric and temporal.

GraphAD supplies the second method ingredient. It links agents in its interaction graph by distance between predicted trajectories, not by visual similarity, and reports a 0.12% collision rate on nuScenes. We borrow its trajectory-distance features as inputs to $g_p$. GraphAD's graph, however, has no relation labels, a closed set of classes, and no way to say "I cannot verify this."

VTransE and Tensor Composition Net pursue the same compositional goal of separating predicate from entity. They do it inside learned 2D appearance embeddings. We pursue it through a physical constraint: the motion scorer has no access to class.

AutoDrive-P³ and AutoDrive-R² are the nearest competitors from the driving-VLM side. Both use free-text chains of reasoning trained with reinforcement learning. Both acknowledge that a hallucinated perception step corrupts everything downstream. Our module is meant to replace that perception step's relational claims with verified, abstainable ones.

Research Program

We first build a relational triplet set on nuScenes. We extend the scenario-mining code released with STSBench (ivc-lrp/STSBench) so that it emits per-frame ⟨agent, predicate, agent⟩ labels for roughly a dozen motion predicates, derived from annotated 3D tracks and HD maps. From these labels we define a compositional split. For each predicate we hold out entire subject-class/object-class pairs, chosen to be the rarest pairs in the training logs. The split therefore measures compositional generalization and long-tail scarcity at once.

Camera-dependent predicates are not derivable from nuScenes labels. Examples are a pedestrian pushing a cart, or a worker signalling traffic. For these we annotate a small evaluation-only set by hand, and we treat this set as secondary and explicitly limited in size.

$g_p$ is a small network over relative position, relative velocity, heading difference, time-to-closest-approach, and GraphAD-style minimum future-trajectory distance. It is trained on tracks from a LiDAR-camera 3D detector and tracker, not on ground truth. $v_p$ runs a frozen open vision-language model with DA-HOI-style one-pass candidate matching.

The reliability estimates are deliberately simple so that they can be audited. $r_L$ is a function of LiDAR point count inside each box relative to the count expected at that range. $r_C$ combines an image-quality score with agreement between the camera-derived 2D box and the projected LiDAR box. The second component is what should catch flat painted "obstacles" of the kind used against vision-only driving models in Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models.

The weights $\alpha_p, \beta_p$ are fit on the training split. We then inspect them, because they give a data-driven answer to which driving relations are geometric and which are visual.

Baselines

The naive reference point is the base vision-language model answering STSnu and our triplet set directly from multi-camera images. We use InternVL 2.5 8B for this, the strongest pixel model in STSBench at 46.07%. The same model is used for $v_p$, which keeps the comparison controlled.

The strongest existing competitor is text-only GPT-4o fed serialized coordinates. STSBench reports 57.08% with ground-truth coordinates. We run it in two forms. The first form uses ground-truth coordinates, as an oracle reference. The second form, which is the fair one, uses the same perception tracks that feed $g_p$, together with the same open-weight language model as our module so that model capacity is matched. Beating the fair version on agent-to-agent STSnu questions, by a margin larger than seed variance, is the bar for the main claim. Matching the ground-truth-coordinate oracle with perception tracks would be a strong result.

The driving-specialist reference points are DriveMM and OmniDrive at their STSBench-reported numbers. For hallucination, we compare against applying RITUAL and TriCD on the base model. RITUAL averages predictions over random image transformations; TriCD contrasts perturbed and saliency-enhanced video passes. Both are given the same base model and the same multi-view input. There is no reported result found on Lacuna for either method on STSnu or nuScenes, so we will run them ourselves.

For trained baselines, we include LoRA fine-tuning of the base model on our training triplets as relational question answering. Its training budget matches the combined training cost of $g_p$ and the weights.

Three ablations isolate the key ingredient. The first gives $g_p$ the two class embeddings as extra input, which re-entangles predicate and appearance. It should match ours on seen pairs and fall behind on held-out pairs. The second removes the reliability gate by fixing $r_L = r_C = 1$. The third removes $g_p$ and lets $v_p$ score motion predicates as well, which tests whether routing, not extra parameters, drives the gain.

Why not just…?

Why not just serialize the LiDAR tracks as text and let the language model reason? This is the real threat, because STSBench's 57.08% shows text coordinates already beat pixels. Our answer has three parts and one experiment. First, the STSBench result used ground-truth tracks, and serialized perception tracks carry detector errors. The language model has no way to discount those errors, so it reads them as fact. Second, serialized text gives no compositional guarantee: the model still sees "bus" and "cyclist" and can apply class priors of the kind documented by POPE for object co-occurrence. Third, serialized text gives no channel for camera-only predicates. The fair serialized-track baseline above settles the question. If it matches us on held-out pairs and under LiDAR degradation, our contribution shrinks to the abstention gate and the appearance predicates, and we will report it that way.

Why not just fine-tune the VLM on driving relational QA? CoLT-Drive shows that LoRA fine-tuning on routine driving data raises routine accuracy from 12.4% to 58.3% but drops rare-case affordance accuracy from 50.3% to 32.4%. STSBench likewise finds that the driving-tuned DriveMM and OmniDrive score below the generalist InternVL. We expect fine-tuning to do well on seen class pairs and poorly on held-out ones. The compositional split measures exactly this.

Why not just fuse LiDAR features into the VLM's visual tokens end to end? This is plausible, but there is no reported result found on Lacuna for a LiDAR-token driving VLM on STSnu. Such a model would still let class appearance and language priors shape motion predictions, and it would offer no explicit signal for when to abstain. Our class-input ablation of $g_p$ approximates the entanglement risk at low cost. A full fused-token model is a stretch comparison if compute allows.

Why not just apply contrastive decoding like RITUAL or TriCD? Both operate on text likelihoods over 2D frames and cost two or three forward passes. Neither has access to metric depth, so neither can reject a relation that is geometrically impossible. Our module adds one cheap track network plus a single matching pass, and only for appearance predicates. We will report accuracy and latency side by side.

Proposed Experiments

The first decisive check is the main result: STSnu accuracy broken down by question category, with the agent-to-agent category carrying the claim, against the baselines above. The second is held-out-pair accuracy on our compositional split, which tests whether the class-free motion scorer actually transfers.

The third check is degradation sweeps on nuScenes validation. We apply camera glare, blur and occlusion, LiDAR beam dropout and range truncation, and both together, reflecting the correlated failures described by Heidecker et al.. The quantities are the hallucination rate, defined as the fraction of confidently chosen hard negatives, and risk-coverage curves. The prediction is that the gated module keeps its error rate flat by abstaining more often, while every baseline's confident-error rate rises. A small set of nuScenes frames with painted flat patterns inserted into the camera views tests the camera-LiDAR agreement term specifically.

The idea fails if any of three things happens. The fair serialized-track baseline matches us on held-out pairs. The class-input ablation does not hurt held-out compositions. Or the gate abstains without reducing confident errors.