Self-driving planners trained on logged data learn to cheat. Most logged driving is straight-line cruising, so a model can score well on the standard error metric, the average distance between its planned path and the human's path, by extrapolating the car's own speed and heading. It can do this while largely ignoring the cameras. The usual fix is to withhold the car's speed and acceleration (its "ego status") from the part of the network that reads the scene. Fairness research learned long ago that withholding a sensitive attribute does not remove it: the model rebuilds it from correlated inputs. A driving model that sees several video frames in a row can infer its own speed from how the world moves past it. We propose to test whether this leak happens in scene-only driving encoders and, if it does, to close it with adversarial debiasing. In adversarial debiasing, a second network tries to predict the forbidden attribute from the model's internal features, and the model is trained so that this second network fails.
The second half of the idea follows from the first. Removing a shortcut will usually make the standard error metric worse, because that metric rewards the shortcut. The strength of debiasing therefore cannot be tuned with the metric the field reports. Safety-critical metric misalignment, the documented gap between displacement error and actual driving safety, is the reason the shortcut persists. It is also the reason any fix for it will look like a regression. We therefore treat debiasing and safety-aligned model selection as a single problem.
Core Claim
Removing ego status from a planner's scene encoder, as dual-branch designs do, is "fairness through unawareness." It leaves ego motion recoverable from temporal visual features. Knowledge distillation from a teacher that sees ego status can put it back. An adversary trained on the scene features to regress ego velocity, acceleration and yaw rate, with its gradient reversed into the encoder, should remove this residual channel. The explicit kinematic branch keeps kinematics available to the planner. We predict that this improves curb-violation and collision behaviour, closed-loop driving and robustness to corrupted speed readings. We also predict that it raises open-loop L2 on straight driving. If our hypothesis is right, that L2 increase is the shortcut being removed. It is not damage to the model.
What Prior Work Establishes
Three papers motivate the gap. Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? shows the following on nuScenes: 73.9% of driving is straight; a camera-free MLP using only velocity, acceleration and yaw beats UniAD on L2 (0.35m vs 0.46m); and planners barely change when perception is blanked to 0.0 mAP. It also introduces Curb Collision Rate (CCR), the fraction of plans that cross a road boundary. The paper is diagnostic only and proposes no training fix. Measuring What Matters reports that ADE correlates with closed-loop driving performance at only r = 0.2609. Accurate Forecasts Do Not Ensure Safe Decision finds that the most accurate pedestrian forecaster (NMRF, minADE 0.19) induces more collisions (4.4%) than a much less accurate one (SocialGAN, minADE 0.61, 3.2%). Together these papers say the metric used to select models actively favours shortcut-reliant ones.
The closest existing method is AdaptiveAD. It keeps ego status out of the bird's-eye-view scene encoder, uses a separate lightweight ego branch, and fuses the two. It reports 22% lower L2 and 57% lower collision rate than VAD, and stability under ego-velocity noise where baselines' error rose by over 800%. AdaptiveAD omits the shortcut from the input but never measures whether the scene features still encode ego motion. It also repairs the scene branch's motion blur by distilling from an ego-informed teacher, which is a plausible route for the shortcut to return. Our proposal differs in two ways: it measures that residual leakage, and it removes it with a training objective rather than an architectural split alone.
The method comes from adversarial debiasing. The gradient-reversal minimax setup used by Xia et al. removes dialect cues from toxicity classifiers. MABR places bias adversaries at several encoder depths, because bias re-emerges in intermediate layers. It reports a 10.6% reduction in the true-positive-rate gap on BERT sentiment at a 0.5% accuracy cost. Two findings constrain the design. First, KANFace finds that high-dimensional features (4096-d) retain more demographic bias than low-dimensional ones (512-d), which warns that wide bird's-eye-view tensors may resist a single adversary. Second, blind erasure causes the familiar accuracy trade-off. That trade-off is why we erase ego status only from the scene pathway and never from the ego branch.
Injecting Planning-Awareness into Prediction and Detection Evaluation supplies the evaluation lens. Its planning-informed displacement metrics (piADE/piFDE) weight each error by how much it would change the planner's cost. In a simulation test, a 0.5m error veering into the ego path scored 25% worse than the same error veering away. We use these metrics for model selection, not as a training loss.
Research Program
The first phase is a leakage audit, and it can end the project. We take trained UniAD, VAD and AdaptiveAD checkpoints and freeze them. We then fit linear and small-MLP probes that decode ego velocity, acceleration and yaw rate from their scene-pathway features at several depths. For AdaptiveAD we probe both before and after its distillation step. As a reference point, we train the same probes on a scene encoder given a single frame, which cannot infer motion from successive frames. If the multi-frame scene-only features decode ego speed barely better than this reference, unawareness already works. In that case the adversarial component is unnecessary, and we would redirect the project toward the selection problem alone.
If leakage is present, we train the debiased model. The scene encoder $E_s$ maps multi-frame camera input $x$ to features $z$. A kinematic branch receives ego status $e$ directly. The planner $P$ outputs a trajectory from both. Adversaries $D_\ell$ attached at several scene-encoder depths $\ell$ try to regress $e$ from the features $z_\ell$ at that depth. The training objective is
$$\min_{E_s, P}\ \max_{D}\ \ \mathcal{L}_{\text{plan}}\big(P(E_s(x), e), \tau\big)\ -\ \lambda \sum_{\ell} \mathcal{L}_{\text{reg}}\big(D_\ell(z_\ell), e\big),$$
where $\tau$ is the logged human trajectory. $\mathcal{L}_{\text{plan}}$ is the usual imitation loss. $\mathcal{L}_{\text{reg}}$ is the adversaries' regression error. $\lambda$ sets how hard the encoder is pushed to hide ego status. We implement the objective with gradient reversal, following the multilayer placement of MABR.
Past bird's-eye-view frames are still warped into the current frame using the car's recorded motion. This warping uses ego motion geometrically but does not store it as a feature. It is the part of motion compensation we keep. We also test a variant in which the adversary sees the distillation target. This checks whether distillation specifically reintroduces the shortcut.
The third phase addresses selection. We sweep $\lambda$ and select it three ways: by open-loop L2, by CCR plus piADE on a held-out nuScenes split, and by closed-loop score on a held-out simulator split. We then ask which selection rule best predicts closed-loop performance on the test routes. We expect L2-based selection to choose $\lambda \approx 0$.
Baselines
The naive reference points are VAD and UniAD trained normally with ego status, together with the camera-free Ego-MLP. On nuScenes average L2, Is Ego Status All You Need reports 0.35m for Ego-MLP and 0.46m for UniAD. Ego-MLP is the floor that any open-loop result must be read against. A method that "beats" it on L2 has only shown that it cruises straight well.
The strongest existing method is AdaptiveAD. The file reports only relative figures for it: 22% lower L2 and 57% lower collision rate than VAD, with stability under velocity noise. We found no absolute NAVSIM or Bench2Drive score for AdaptiveAD on Lacuna. We will therefore reproduce it ourselves from the same VAD backbone, with the same training data, schedule and hyperparameter-search budget as our model. Our model is AdaptiveAD plus adversaries, so this is a clean like-for-like comparison.
To beat AdaptiveAD, the debiased model must lower CCR and collision rate on nuScenes turning and intersection scenes, and must improve the closed-loop NAVSIM and Bench2Drive scores. It must also avoid losing those gains under ego-velocity noise. Each improvement must be statistically separated across three training seeds. A higher straight-road L2 is acceptable.
The key ablation keeps the dual-branch architecture and training budget fixed and varies only the adversary. The comparison conditions are: no adversary; an adversary regressing a random, shuffled copy of ego status, which controls for the adversary merely acting as a noise regularizer; and a single final-layer adversary versus adversaries at several depths. A fourth baseline, ego-status dropout and noise augmentation on a standard single-branch model, represents the cheapest possible intervention. A fifth trains VAD with a planning-sensitivity-weighted loss, using piADE-style weights. It represents the direct metric-alignment route.
Why not just…?
Why not just omit ego status from the scene encoder, as AdaptiveAD does? This alternative is the main threat, and phase one is designed to settle it. If probes cannot decode ego motion from AdaptiveAD's scene features beyond the single-frame reference, omission suffices and we will say so. The need for AdaptiveAD's ego-informed distillation to fix motion blur is circumstantial evidence that the scene branch is being pushed back toward ego information. That is only a hypothesis until the probes are run.
Why not just train with a safety-weighted loss, such as inverse time-to-collision weighting (Zhan et al.) or planning-sensitivity weighting? Reweighting changes which errors matter but not which inputs the model uses to reduce them. When most samples are straight cruising, inertia remains the cheapest way to lower even a reweighted loss. This is an argument, not a result. The weighted-loss baseline, combined with the camera-blanking test (planning should degrade when the cameras are removed), measures whether reweighting alone restores reliance on the scene.
Why not just fix decisions post hoc, like the risk aligner of Accurate Forecasts Do Not Ensure Safe Decision? That aligner cut collisions by about one-third on frozen pedestrian forecasters, but its authors report it becomes overly conservative and needs an extra progress heuristic. More fundamentally, re-ranking outputs cannot help a planner whose outputs never depended on the camera. The two approaches are complementary rather than competing, and we do not claim otherwise.
Why not just rebalance the data toward turns? Rebalancing is cheap and we include it as a variant of the dropout baseline. However, the leak we target comes from ego motion that is visible in video, and that signal exists at any data mix. Near-collision interactive data also cannot be collected safely at scale.
Decisive Checks
The experiments run on nuScenes, which provides L2, collision rate and CCR, stratified into straight and turning scenes and scored with piADE. Closed-loop evaluation uses NAVSIM and Bench2Drive, with the ego-velocity noise injection that AdaptiveAD used, plus camera blanking.
Three outcomes decide the proposal. The first is whether probe R² for ego velocity on AdaptiveAD's scene features is meaningfully above the single-frame reference. If it is not, the adversarial idea is dropped. The second is whether the adversarial model beats the shuffled-target control on CCR and closed-loop score. If it does not, any gain was generic regularization. The third is whether selecting $\lambda$ by CCR plus piADE predicts closed-loop ranking better than selecting by L2. If it does, the paper's second claim holds: debiasing a shortcut cannot be evaluated with the metric that created it.
References
- Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
- Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving
- Accurate Forecasts Do Not Ensure Safe Decision
- Decoupling Scene Perception and Ego Status: A Multi-Context Fusion Approach for Enhanced Generalization in End-to-End Autonomous Driving
- Demoting Racial Bias in Hate Speech Detection
- MABR: Multilayer Adversarial Bias Removal Without Prior Bias Knowledge
- Investigating bias in deep face analysis: The KANFace dataset and empirical study
- Injecting Planning-Awareness into Prediction and Detection Evaluation
- Towards a Fatality-Aware Benchmark of Probabilistic Reaction Prediction in Highly Interactive Driving Scenarios