Programs that test self-driving software by generating dangerous traffic have learned to keep their crashes honest. They now check, with a kinematic safety model, that a careful driver could have avoided each crash, so a collision reflects a real weakness in the software and not a physically impossible setup. Their output is still only "the car crashed here, and the crash was avoidable." An engineer receives no information about which part of the driving software failed: whether it never noticed the cyclist, noticed it too late, or noticed it and chose the wrong maneuver. Medical diagnostic AI addressed a version of this problem with concept bottleneck models. These are networks that must first predict human-readable intermediate findings, such as "spiculated margin" or "joint space narrowing," before giving a diagnosis. Because those findings are explicit, a clinician can correct one at test time and see whether the diagnosis changes.
We propose to carry that test-time correction into closed-loop driving simulation and use it as a diagnostic instrument. The driving policy routes its decisions through a small set of simulator-checkable driving concepts, such as "vehicle in ego lane within 20 m," "pedestrian entering crosswalk," or "occluded actor near the conflict point." For every crash that a feasibility-constrained adversary produces and that a safety model certifies as avoidable, we re-simulate the episode with individual predicted concepts replaced by their ground-truth values from the simulator. The concepts whose correction prevents the crash form the crash's diagnosis. Crashes that no concept correction prevents are, by elimination, failures of the decision layer. Each avoidable crash then comes with a short report that can be checked, and the scenario generator can be pointed at a chosen failure class.
Core Claim
Fidelity in adversarial scenario generation is currently defined relative to an idealized driver: an avoidable crash under a kinematic safety model. We argue that a second, policy-relative notion is both computable and more useful: whether the crash is avoidable by this same policy given correct concepts. This notion splits certified-avoidable crashes into two groups. Concept-rescuable failures are perception or understanding errors, and each is attributed to specific concepts. Concept-irreducible failures are planning errors. The split exists only when the policy has an explicit, intervenable bottleneck and only when interventions run in closed loop, because a corrected concept changes the ego trajectory and therefore everything the ego observes afterwards. Our claim is that this triage recovers planted faults reliably. We also claim that repairing the policy according to the triage (relabeling concept data for concept failures, adversarial fine-tuning of the planner for irreducible ones) reduces held-out avoidable crashes more per unit of training than generic adversarial fine-tuning.
Position Relative to Prior Work
Two lines of work motivate the gap. The first is feasibility-constrained generation. FREA uses Hamilton-Jacobi reachability, a method for computing which states can still escape a collision, to penalize adversaries that make crashes inevitable. ScenePilot shields adversaries with Mobileye's Responsibility-Sensitive Safety (RSS) feasibility score so they stay in a "boundary band" of solvable-but-hard scenarios. Both treat the ego stack as opaque, and their notion of avoidability is purely kinematic. We reuse ScenePilot's RSS score as the first filter and add the policy-relative second test.
The second line is interpretable control under stress. Driving through the Concept Gridlock puts a CLIP-based concept bottleneck into a driving model, but evaluates it only open-loop on recorded logs and never intervenes. MA²T studies how pixel and feature noise propagates through tracking, mapping, and prediction modules of end-to-end stacks and finds the tracking–prediction interface most fragile. Its probes are norm-bounded noise rather than physically plausible traffic, and its intermediate tensors are not human-readable.
The method itself comes from Concept Bottleneck Models, which showed that correcting two concepts on knee radiographs cut task error from over 0.4 to about 0.3. We turn intervention from a usability feature into a causal test. Medical multi-task systems such as PN-SAMP and two-stage retinopathy grading in Lesion detection and Grading of Diabetic Retinopathy supply the design pattern of grounding the final decision in explicit intermediate findings.
Two papers constrain the design. Integrating Clinically-Relevant Features into Skin Lesion Classification found that forcing hand-crafted clinical features into the model hurt the nevus class. Concept Gridlock found driving performance saturates near 100 concepts. Together they warn that a bottleneck can cost task performance, so we measure that cost explicitly against a matched black-box policy rather than assume it is free. Causal generators such as CausalAF and Causal Bayesian Networks apply causal structure to the environment. Our causal structure is on the ego side, which is the missing half that the research file's Gap 5 identifies.
Research Program
The policy. We build an intervenable driving policy in CARLA. A perception network maps camera and LiDAR input to roughly 40–80 binary or ordinal concepts, each computable exactly from CARLA's privileged state. A temporal planner maps the concept history to control. Concepts are deliberately simulator-groundable rather than CLIP text prompts, because ground truth must be available for every intervention. Concepts are hard-thresholded at the bottleneck to limit "leakage," the case where soft concept scores carry hidden information that the planner exploits and that would make interventions misleading. We report a soft-concept variant to measure how much leakage matters.
The generator. Scenarios come from a ScenePilot-style RSS-shielded adversary on SafeBench routes. Each collision is first checked for RSS feasibility, and crashes with RSS score below zero are discarded as impossible.
The triage. For an avoidable crash in scenario $s$, let $\hat c_t$ be the predicted concept vector at step $t$, $c^*_t$ the ground truth, and $\mathrm{crash}(s;\, k)$ the outcome of re-simulating $s$ with concept $k$ replaced by $c^*_{t,k}$ at every step. The diagnosis is the minimal rescuing set
$$ D(s) = \text{smallest } K \subseteq \{1,\dots,m\} \text{ such that } \mathrm{crash}(s;\, K) = 0, $$
found greedily: first single concepts, then pairs, ranked by how often the prediction disagrees with ground truth. If even $K=\{1,\dots,m\}$ (full oracle concepts) does not rescue the episode, the crash is labeled planner-irreducible.
Two replay modes matter, and we report both. In one, the adversary replays its recorded actions. In the other, the adversary reacts to the corrected ego. A diagnosis that holds only under replay is flagged as fragile. We also record the first time step at which the intervention must begin for the rescue to work, which yields statements of the form "the occluded-actor concept was wrong from 1.4 s before impact."
Targeted generation. The adversary's reward adds a term for producing avoidable crashes of a requested triage class, for example "irreducible" or "rescued by the occlusion concept." Engineers can then stress a specific weakness, much as a clinician orders a specific follow-up test.
Repair. Concept-attributed failures trigger relabeled perception training on frames near the misjudged concepts. Irreducible failures trigger planner fine-tuning on those scenarios.
Baselines
The naive reference point is ScenePilot-style generation with no attribution at all, followed by the standard repair of adversarially fine-tuning the whole policy on all collected boundary-band crashes. This is the strongest existing method on the question "does the stress test make the car safer." ScenePilot reports a mean collision rate of 0.893 on SafeBench, against 0.831 for ChatScene and 0.586 for AdvSim, and reports that boundary-band fine-tuning dropped held-out crash rate from 85.4% to 7.2%. We will rerun it against our own policy, not compare to those numbers directly. It gets the same SafeBench routes, the same scenario budget, and the same number of fine-tuning gradient steps as our triage-guided repair. Triage-guided repair beats it if it reaches a lower held-out avoidable-crash rate at matched steps, or the same rate in clearly fewer steps, across at least three seeds. With ScenePilot already at 7.2%, a ceiling effect is a real possibility. We therefore also report the crash rate on a harder held-out set generated by a fresh adversary trained against the repaired policy, where headroom remains.
The second baseline is a matched black-box policy with the same perception backbone and planner capacity but no bottleneck, explained post hoc with Integrated Gradients and attention maps. It answers whether the bottleneck is needed for diagnosis at all. It also tells us what the bottleneck costs in clean SafeBench driving score and route completion. For the overall SafeBench score, the only reference in the file is a top of 0.622 for PPO with 4D input, falling to 0.508 under adversarial trajectory optimization.
The third baseline is oracle-module substitution in the style of MA²T's module-level analysis. We use a modular version of the stack and swap whole modules (detection, tracking, prediction) for ground truth. This gives coarse attribution without concepts.
The key ablation is open-loop intervention. Concepts are corrected and the planner's action is recomputed on the recorded observation sequence without re-simulating. If open-loop triage recovers planted faults as well as closed-loop triage does, the closed-loop claim fails. A second ablation replaces the RSS filter with no filter, which shows how much of the irreducible category is filled with impossible crashes.
Why not just…?
Why not just apply saliency maps to a black-box policy? Saliency shows where the network looked, not what it would have done had it seen correctly. It also cannot be re-simulated, so it supports no counterfactual. Concept Bottleneck Models argues post-hoc explanations do not reflect the computation and permit no intervention. The planted-fault experiment below tests this directly by asking whether saliency places mass on the blinded object class more often than on others. If saliency recovers planted faults about as well as concept interventions do, that is a negative result for us, and we will report it.
Why not just run the modular stack with ground-truth perception? This is the most serious alternative. For a strictly modular stack, oracle-perception replay already separates perception failures from planning failures, and our module-substitution baseline is exactly this test. What it cannot provide is attribution at the level of which fact was wrong: occlusion versus traffic-light state versus actor intent. It also has no handle on learned end-to-end policies, where no module boundary exists. If concept-level diagnoses add no repair benefit over module-level ones in the repair experiment, the honest conclusion is that module oracles suffice, and the bottleneck's value narrows to readability.
Why not just label failures by scenario type? SafeBench already tags scenarios with NHTSA pre-crash types such as cut-in or unprotected left turn. If the triage class were predictable from scenario type, the diagnosis would add nothing. We will measure how well scenario type predicts triage class. Our hypothesis is that the same scenario type splits across concept-rescuable and irreducible failures, but this is an empirical question that the data will settle.
Proposed Experiments
Planted faults. This is the decisive check. We inject known defects into the trained policy and test whether the triage recovers each one. The defects are: suppressing the pedestrian concept at night, adding a 300 ms delay to the lead-vehicle-braking concept, and corrupting the planner so it ignores a gap-acceptance concept. Metrics are precision and recall of the planted cause among avoidable crashes, compared across closed-loop intervention, open-loop intervention, saliency, and module substitution. No reported result on Lacuna was found for fault-attribution accuracy in driving.
Repair. On held-out SafeBench routes and on a fresh-adversary test set, we compare triage-guided repair against ScenePilot-style uniform fine-tuning at matched compute.
Cost of the bottleneck. We report clean SafeBench score, collision rate, and route completion for the concept policy against the black-box policy.
Scale and transfer. We repeat the triage on MetaDrive with Waymo Open Motion Dataset scenarios, using CAT's generator, which reports 91% attack success against a replay agent. This checks that the triage does not depend on one generator.
The idea fails if any of the following holds: closed-loop triage does not beat open-loop triage on planted faults; most avoidable crashes require large concept sets to rescue, so diagnoses are not crisp; or triage-guided repair is no better than uniform fine-tuning.
References
- FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality
- ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving
- Driving through the Concept Gridlock: Unraveling Explainability Bottlenecks in Automated Driving
- Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving
- Concept Bottleneck Models
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction
- Lesion Detection and Grading of Diabetic Retinopathy via Two-Stages Deep Convolutional Neural Networks
- Integrating Clinically-Relevant Features into Skin Lesion Classification
- CausalAF: Causal Autoregressive Flow for Safety-Critical Driving Scenario Generation
- Risk Scenario Generation for Autonomous Driving Systems based on Causal Bayesian Networks
- CAT: Closed-loop Adversarial Training for Safe End-to-End Driving