Self-driving teams increasingly choose between perception models using synthetic test data: simulator renders, digitally corrupted images, or images from generative models. The working assumption is modest. Synthetic scores need not match real scores, but they should put candidate models in the same order. Recent work supports that assumption for single images. The danger in driving, however, is rarely a single bad frame. A tracking filter absorbs one missed detection. It loses the object when the misses run three or four frames in a row. This proposal asks whether synthetic proxies preserve the ranking of detectors on that quantity: how long each detector stays blind to an object that actually matters for collision avoidance.
The claim is that they often will not, and for a specific, testable reason. Most synthetic perturbations are applied independently frame by frame. Real adverse conditions persist: a dark pedestrian stays dark for the whole approach. Independent perturbations break misses into short runs, so they flatter detectors that average over several frames and hide the long blind spells that actually defeat trackers. If this is right, a proxy can keep a high rank correlation on per-frame accuracy and still invert the ranking that matters for safety. We will measure this effect, check whether it is real or only a restatement of per-frame miss rate, and test a cheap correction that matches the proxy's temporal persistence to a few real anchor models.
Core Claim
We define a persistence-weighted safety metric. For each object inside a dynamics-aware critical zone, it counts the fraction of critical time the object spends inside a miss run longer than the tracker's tolerance. We then make three conjectures. First, on real data this metric ranks detectors differently from per-frame critical recall. Second, synthetic proxies built from frame-independent perturbations preserve the per-frame ranking much better than the persistence ranking. Third, re-sampling the proxy so that its miss runs match those of a handful of anchor models on real data restores the persistence ranking. The first conjecture is a precondition. If real persistence rankings equal per-frame rankings, the temporal ingredient is redundant and we will report that.
Concretely, let $m_{o,t}\in\{0,1\}$ indicate that object $o$ is missed at frame $t$. Let $Z_t$ be the set of objects inside the critical zone at frame $t$. Let $k$ be the tracker tolerance in frames, fixed in seconds and converted per dataset frame rate. The score is
$$S_k = \frac{\sum_{t}\sum_{o\in Z_t} \mathbb{1}[\,o \text{ is in a miss run of length} \ge k \text{ at } t\,]}{\sum_t |Z_t|}.$$
If misses were independent with rate $p$, $S_k$ would be roughly a monotone function of $p$ (it scales like $p^k$), and the ranking would collapse onto miss rate. The metric is informative only to the extent that detectors differ in how strongly their misses cluster in time. That is the empirical question.
Grounding in Prior Work
Several papers motivate the temporal focus. Reliability Validation of Learning Enabled Vehicle Tracking shows that a Kalman-filter tracker absorbs isolated detector failures but loses tracks after 3–4 consecutive misses. Its evidence is aerial imagery at 1.25 Hz. Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles finds that collisions follow the longest non-detection interval rather than error frequency. Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study finds that static evaluation disagreed with closed-loop outcomes in 44% and 34% of scenarios for two steering models, always in the optimistic direction. None of these works asks whether synthetic data reproduces the persistence structure across many models. That ranking question is ours.
The synthetic-proxy literature supplies our method and our null hypothesis. Synthetic data for model selection reweights synthetic samples using about five anchor models and reaches Spearman 0.986 (Diffusion Transformer generator) and 0.978 (BigGAN) against ImageNet rankings. AutoEval Done Right combines many synthetic labels with a few real ones through prediction-powered inference, a statistical correction that keeps metric estimates unbiased while cutting variance. Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions? reports Pearson about 0.795 between Cityscapes-C and real adverse weather, falling to 0.349 for fog. Reliability in Semantic Segmentation: Can We Use Synthetic Data? reaches up to 0.85 with diffusion-generated edge cases. All of these evaluate static frames. We reuse their anchor-calibration and de-biasing ideas, but we calibrate temporal statistics rather than per-image weights.
The closest existing work is Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration. It asks whether synthetic images trigger the same safety-relevant errors as real ones and tunes generator settings to reduce the mismatch on KITTI. It works per image and uses box area as its safety notion, which its authors call an oversimplification. We differ on both axes: our unit is the error run, and our safety notion is dynamics-aware.
The safety-metric literature supplies our critical zone and our downstream checks. Interaction-Dynamics-Aware Perception Zones computes zones from reachability analysis, which asks which nearby agents could reach the ego vehicle given velocities and braking limits. It flags 0.75% of false positives as critical versus 6.6% for a circular zone. Risk Ranked Recall supplies risk tiers. It also delivers a warning: zero imminent-collision objects in 24 Waymo sequences. That scarcity constrains our statistics. Learning to Evaluate Perception Models Using Planner-Centric Metrics supplies a planner-based score, which measures how much a learned planner's trajectory distribution shifts when fed predicted rather than true detections. We use it as an independent downstream reference, while noting its own planner bias.
Research Program
We first build a real-data reference ranking. A pool of roughly 15–20 detectors is run on nuScenes (nuScenes). The pool is chosen to vary temporal aggregation: single-frame camera models, multi-sweep LiDAR models, and temporal camera models. The nuScenes paper reports that stacking up to 10 LiDAR sweeps markedly helps PointPillars, so sweep count is a natural dial for temporal aggregation. We also use the 142 critical pedestrian tracks in Argoverse 1.1 identified in Towards Safety-Aware Pedestrian Detection in Autonomous Systems, which offer a higher frame rate. Every detector's outputs feed one fixed Kalman tracker. For each detector we record four quantities: per-frame critical recall, $S_k$, tracker track-loss and velocity-error rates on critical objects, and the planner-based score.
Second, we build proxies that differ along a single axis: whether the perturbation is independent across frames or persistent across frames. The first proxy family applies corruptions to real nuScenes sequences. In the independent version, corruption parameters are resampled every frame. In the persistent version, parameters evolve slowly and occlusion masks stick to objects. The second family is CARLA-rendered sequences of comparable scenarios. The third is an independent-frame diffusion-edited set in the style of Reliability in Semantic Segmentation: Can We Use Synthetic Data?. For each proxy we compute every metric and its Spearman and Kendall correlation with the real ranking, with confidence intervals from bootstrapping over scenes.
Third, we test a temporal calibration. We take a few anchor detectors and fit a two-state hidden Markov model to their real miss sequences, which captures how likely a miss is to follow a miss. This follows the perception error models of On the Simulation of Perception Errors in Autonomous Vehicles. Those models inject errors onto ground-truth boxes. We use the fitted persistence instead to set how slowly the proxy's corruptions evolve, or to re-sample proxy sequences so that the anchors' run-length distributions match. Non-anchor detectors are then ranked on the calibrated proxy.
Baselines
The naive reference is ranking detectors by per-frame critical recall or by the nuScenes detection score on the uncalibrated synthetic proxy, with no temporal or anchor information. The nuScenes paper reports a Megvii baseline detection score of 63.3% on real data, a useful sanity anchor. This baseline shows how far a standard benchmark gets, and it must be beaten on the persistence ranking specifically, not on per-frame ranking.
The strongest existing method is anchor-based sample reweighting from Synthetic data for model selection. It fits per-sample weights by ridge regression so that synthetic scores of anchor models match their real scores. It reports Spearman 0.986 (Diffusion Transformer) and 0.978 (BigGAN) on ImageNet. No reported result found on Lacuna for this method on driving sequences or temporal metrics. It is the strongest competitor because it uses exactly our anchor budget and may already absorb temporal mismatch indirectly. We run it with the same anchor detectors, the same proxy data, and the same hyperparameter search budget. We apply it twice: once fit to per-frame recall and once fit directly to $S_k$. Beating it means a statistically higher Kendall correlation on $S_k$ for held-out detectors, with non-overlapping bootstrap intervals.
The second strong competitor is prediction-powered inference from AutoEval Done Right. Here each detector's $S_k$ is estimated from the proxy plus a small labeled real subset matched in size to the anchor-fitting data. That paper reports a 20–50% gain in effective sample size. We compare at matched numbers of real labeled sequences. The safety-aware fidelity calibrator of Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration is included as a generator-tuning baseline. It reports 105 versus 123 inconsistent predictions for its best and worst settings, but no ranking correlation.
Two ablations isolate the key ingredient. The first swaps $S_k$ for per-frame critical recall while keeping the same reachability zone, which tests whether persistence adds anything beyond miss rate. The second swaps reachability zones for circular zones, which tests whether the safety weighting matters.
Why not just…?
Why not just rank by per-frame critical recall on synthetic data? If misses cluster about equally for all detectors, $S_k$ is a monotone function of miss rate, and this simpler approach is enough. This is the most serious threat to the proposal. The first ablation settles it directly. On real data alone, if Kendall correlation between the recall ranking and the $S_k$ ranking is near one, and both predict track loss equally well, we drop the temporal metric. In that case the contribution reduces to validating proxies under safety-zoned recall. The prediction that multi-sweep detectors convert scattered misses into fewer but longer runs is what gives us reason to expect the rankings to differ.
Why not just apply the anchor reweighting of Shoshan to the temporal metric? Reweighting can only up-weight sequences the proxy already contains. If independent perturbations never produce long miss runs, no choice of weights can create them. The persistent-corruption arm tests this. If reweighting the independent proxy matches the calibrated persistent proxy, our calibration step is unnecessary, and the result would be reported as such.
Why not just run everything closed-loop in a full simulator? Closed-loop simulation is the eventual target. But On the Simulation of Perception Errors in Autonomous Vehicles shows that full sensor simulation is too costly for hundreds of runs per scenario. Simulator rankings are themselves a synthetic proxy whose real-world validity is unverified. CARLA is therefore one of our proxies, not the ground truth.
Why not just use 2D corruption benchmarks as they are? Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions? already shows weak fog and night alignment for single frames. Our independent-versus-persistent comparison tests a separate failure mode that would remain even if per-frame appearance were perfect.
Proposed Experiments
The first decisive check is the precondition on real nuScenes and Argoverse. We measure whether the $S_k$ ranking and the critical-recall ranking diverge (Kendall correlation clearly below one, with bootstrap intervals). We also measure whether $S_k$ predicts the fixed tracker's track-loss ranking and the planner-based ranking better than recall does. The second check is the core prediction. On the independent-frame proxies, correlation with the real ranking should stay high for recall but drop for $S_k$. The drop should show multi-frame detectors systematically over-ranked, and it should shrink in the persistent-corruption arm. The third check is calibration. With about five anchors, the persistence-calibrated proxy should beat anchor reweighting and prediction-powered inference on held-out detectors' $S_k$ ranking at matched real-label budgets.
Because risk-ranked recall found no imminent-collision objects in real Waymo data, we report results at the potential-risk tier from reachability zones. We state explicitly that the highest-risk tier cannot be validated against real data here. Any of three outcomes would be informative: proxies already preserve persistence rankings, temporal calibration repairs them, or persistence collapses onto miss rate. Each changes how synthetic data should be used for safety-relevant model selection.
References
- Reliability Validation of Learning Enabled Vehicle Tracking
- Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles
- Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study
- Synthetic data for model selection
- AutoEval Done Right: Using Synthetic Data for Model Evaluation
- Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions?
- Reliability in Semantic Segmentation: Can We Use Synthetic Data?
- Instance-Level Safety-Aware Fidelity of Synthetic Data and its Calibration
- Interaction-Dynamics-Aware Perception Zones for Obstacle Detection Safety Evaluation
- Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles
- Learning to Evaluate Perception Models Using Planner-Centric Metrics
- nuScenes: A multimodal dataset for autonomous driving
- Towards Safety-Aware Pedestrian Detection in Autonomous Systems
- PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles