Do Privacy–Utility Trade-offs Found in Simulation Survive Real Roads?

Self-driving cars that cooperate share compressed internal features of what their LiDAR sensors see, so that each car can detect objects hidden from its own view. These features can also leak the raw scene, including bystanders, driveways, and fine geometry, to any receiver or eavesdropper. The usual defense is to clip the features and add noise before sending them. Choosing how much noise, how much compression, and which channels to share is a trade-off between privacy and detection accuracy. Methods that map this trade-off automatically assume the setting where they are tuned is the setting where they will run. In cooperative driving, that tuning setting is almost always a simulator. We know from perception alone that the simulator flatters models: an intermediate-fusion detector that is strong on simulated data drops to 33.4% [email protected] on real driving data (S2R-ViT).

This note proposes two things. The first is to measure whether a privacy–utility Pareto front found in simulation still describes reality. A Pareto front is the set of settings where privacy cannot improve without losing accuracy. The second is to build a cheap correction for when it does not. The correction keeps most of the search in simulation and spends a small budget of real, labeled evaluations to re-rank the candidate settings. The test bed pairs the simulated cooperative-perception dataset OPV2V with its real-world counterpart V2V4Real. Both are already used by the cooperative-perception systems this work builds on.

Core Claim

The claim has two parts and can fail in an informative way. The first part is that sim-to-real shift does more than move the whole front down. It reorders it. Settings that sit at the "knee" of the simulated front, where a small privacy gain stops costing much accuracy, will not be the knee in reality. Two mechanisms drive this. Real sensor statistics change how much detection accuracy survives a given amount of noise. Real communication delay, which V2V4Real reports degrades intermediate fusion even at 100 ms, compounds with the injected privacy noise rather than simply adding to it. The prediction is that a simulator-chosen operating point, evaluated on real data, will often be beaten on both privacy and accuracy at once by some other configuration.

The second part is constructive. If the reordering is real, a two-level search can recover most of the real front's hypervolume using a few real evaluations. Hypervolume is the area of objective space dominated by a front, so larger is better. In this search, a Gaussian-process surrogate (a probabilistic regression model that also reports its uncertainty) treats each simulated measurement as a cheap, biased reading and learns a correction from a small number of real ones. If instead the sim front transfers with a preserved ranking, that is also a useful result. It would license simulation-only privacy tuning for this domain, which no current paper can justify.

Research Program

The system under study is a cooperative LiDAR detector with intermediate fusion. We use the attentive fusion architecture from OPV2V and CoBEVT as evaluated in V2V4Real, with a release mechanism inserted on the shared bird's-eye-view feature map. A bird's-eye-view map is a top-down grid of learned features. The release mechanism has four knobs: a per-cell clipping threshold, a Gaussian noise scale, a compression ratio, and a channel-retention mask. The clipping and noise together give a formal per-frame differential-privacy budget ε through the Gaussian mechanism. ε is a domain-independent number, and that independence is the point. ε stays fixed across sim and real, while the accuracy and the empirical leakage at that ε do not.

Three objectives define the front. Utility is detection [email protected] on the receiving car. Empirical leakage is measured by an inversion attacker, a decoder trained to reconstruct the sender's raw point cloud and fine-grained occupancy from the released features, scored by occupancy IoU at a resolution finer than the detection grid. This targets the raw-sensor-exposure risk that STAMP names but leaves to architectural isolation. Communication cost is bits per frame. Every configuration is evaluated in two settings: the simulated one, and the real one under recorded asynchrony and localization error. The attacker is retrained in each domain, so that leakage is measured against an adversary fitted to that domain's data.

The methodological backbone comes from DPareto. DPareto models privacy loss and empirical error of DP-trained networks as black boxes, using Gaussian-process surrogates and a hypervolume-based acquisition rule. It reports that the surrogate costs under 0.1% of training time and that it beats random and grid search in the high-privacy regime. We reuse its machinery but change the question. DPareto assumes the evaluation oracle is the deployment oracle. We give the surrogate two oracles: simulation, which is cheap and plentiful, and real data, which is limited by labeled real frames and per-configuration fine-tuning. The surrogate is a multi-fidelity model in which the real objective equals a learned transformation of the simulated objective plus a residual, fit jointly. The acquisition rule spends real evaluations only where the residual's uncertainty threatens which points end up on the real front.

Two further contributions explain the transfer behavior. We decompose the real-front shift into a sensor-statistics part and a deployment part (latency and pose jitter). To do this we evaluate on real data with timestamps realigned using ground-truth poses versus with the raw recorded asynchrony. We also test the tension raised by On privacy preserving data release of linear dynamic networks. That paper optimizes a three-way trade-off of privacy, controllability, and perturbation cost in a simulated HVAC model, and notes that its guarantees break under unmodeled dynamics and informed adversaries. Here we check empirically whether leakage at fixed ε rises on real data because the real scene has more structured, more predictable content for an attacker to exploit.

Baselines

The naive reference point is the default practice of picking one configuration in simulation by a weighted sum of accuracy and leakage and deploying it unchanged. That chosen point, evaluated on real data, is the "do nothing" bar. Next comes uniform noise with no compression or channel selection, swept over noise scale. This mirrors the heuristic obfuscation that When Visual Privacy Protection Meets Multimodal Large Language Models found inferior to Pareto optimization, and the uniform budget allocation that FDPEL improved on for medical tabular data. Here we test that ordering in a new domain rather than assume it.

The strongest existing method is DPareto run in simulation only, with its full budget, and its front then evaluated on real data. A second strong competitor is DPareto run directly on real data with the same small real budget our method gets, and no simulated data. The two-level method must beat both on real-front hypervolume at equal real-evaluation budget and equal total GPU hours. All arms share the same detector backbone, attacker architecture and training schedule, and real calibration split. Final scoring uses a held-out real test split that no arm sees during search.

For accuracy, the reference numbers are 66.5% [email protected] for CoBEVT trained and tested in the V2V4Real benchmark, and 44.1% [email protected] for the domain-adapted S2R-ViT. These define the no-privacy ceilings for real-trained and sim-trained-then-adapted models. No reported result was found on Lacuna for feature-inversion leakage of cooperative-perception features, or for any privacy–utility hypervolume on OPV2V or V2V4Real. DPareto also reports no hypervolume number on its own benchmarks. We will therefore establish these figures ourselves.

The ablation that isolates the key ingredient is the two-level surrogate with the learned sim-to-real transformation replaced by an identity map. That variant treats real points as extra samples of the simulated function. If it matches the full method, the correction model is doing nothing and the gain comes only from having some real data.

Why not just…?

Why not just run the Pareto search directly on real data? V2V4Real exists, so this is a serious threat. If real labeled frames and per-configuration fine-tuning were cheap, direct real search would be the right answer and our method would be pointless. The proposal answers it with a budget sweep. We vary the number of real evaluations from very few up to the full DPareto budget, and report where direct real search catches up. If it catches up at around five real evaluations, we will say the simulator adds nothing for privacy tuning here. That would be a finding worth publishing, but it would not support the method.

Why not just inject realistic latency and pose jitter into the simulator? This is the other real threat. S2R-ViT shows deployment noise is a large part of the gap, and OpenCDA can simulate delay. We include a "deployment-noise-augmented simulation" arm. The decomposition experiment shows how much of the front shift it removes. If it closes most of the gap, the conclusion becomes that privacy fronts should be tuned in deployment-aware simulation, and the calibration machinery becomes secondary. We expect a residual from sensor statistics, given that domain adaptation recovers only an average 7.46% of the gap in V2V4Real, but the experiment decides.

Why not just apply domain adaptation first and then tune in simulation? S2R-ViT's adversarial feature alignment makes simulated and real features look alike. One might expect sim-tuned privacy settings to then transfer. But alignment changes exactly the statistics an inversion attacker exploits. Making features domain-invariant could raise or lower leakage, and nobody has measured which. We run the full pipeline on top of S2R-ViT's alignment as a separate arm and report whether front reordering persists after adaptation.

Why not just rely on frozen encoders and adapters, as STAMP does? STAMP protects model weights, not the content of what is shared. EgoPrivacy shows that foundation models recover identity and demographics with about 80% accuracy from high-level visual signals alone, without access to any weights. We include STAMP-style adapter features with no noise as a leakage reference point, so this is measured directly rather than argued.

Proposed Experiments

The first decisive check is rank preservation. We evaluate about a hundred configurations, sampled to cover the knob space, in both domains. We report Kendall rank correlation between simulated and real values of each objective. We also report the fraction of simulated-front points that remain non-dominated on real data. A correlation near one together with a stable front membership would falsify the first claim.

The second check is the knee test. We locate the knee of the simulated front and ask whether any real-evaluated configuration dominates it on both accuracy and leakage. We report the size of that domination, in AP points and IoU points.

The third check is the main comparison. We plot real-front hypervolume, on the held-out real split, against the number of real evaluations for five arms: simulation-only DPareto, direct real DPareto, deployment-noise-augmented simulation, the two-level method, and its identity-map ablation. The two-level method counts as succeeding if it reaches at least 90% of the hypervolume that direct real search attains at its full budget, while using a small fraction of that budget, and if it beats the identity-map ablation.

The fourth check is the decomposition and the fixed-ε leakage comparison. Holding ε constant, we compare inversion IoU between simulated and real data, with and without timestamp realignment. This shows whether formal guarantees under-describe real leakage, and whether delay or sensor statistics drives the shift. All results are reported with confidence intervals over attacker retraining seeds, since a single attacker run can understate leakage.