Synthetic Users on Real Apps: Measuring How Far Simulated Traffic Can Replace Real Traces for Plaintext Leak Detection

Phones and smart-home devices still send sensitive data, such as GPS coordinates, device IDs and status messages like "Nobody is home", over unencrypted connections. Spotting that a coordinate left the device is easy. The hard question is whether the transmission was legitimate (a weather widget the user can see) or covert (an ad library sending the same coordinate to the same cloud provider). Detectors that make this call need labeled real traffic. Real traffic is expensive to collect, slow to label, and dangerous to share, because it contains the very leaks it documents. We propose to build that training data from synthetic users instead. Scripted personas drive real, unmodified apps on instrumented devices. Each persona follows a fabricated life: a home, a workplace, routines, and planted "canary" values that appear nowhere else. Every captured flow therefore comes with an exact label and contains no real person's data.

The scientific contribution is measurement rather than the generator itself. We ask how many synthetic flows are worth one real labeled flow for training a leak-legitimacy detector. We also ask what fraction of real data can be replaced before the detector fails on traffic from real people, and which property of the synthetic users controls that exchange rate. Synthetic-data research has measured these quantities for video tracking but never for network privacy monitoring. Generative traffic models, the obvious alternative source of synthetic traces, have been shown to regurgitate the identifiers they were trained on.

Core Claim

Our claim is that, for deciding whether a plaintext flow is legitimate, what matters is coverage of the pairings between what the user sees and what the app sends. Byte-level realism matters much less. A corpus produced by real apps under behaviorally diverse synthetic personas should substitute for a large, measurable fraction of real labeled traffic. It should do so with zero exposure of real users, and diversity of persona behavior should matter more than corpus size.

The claim rests on three findings. On the motivation side, Chang et al. used a procedural video generator to show that synthetic frames can replace 60–80% of real data at roughly a 30:1 frame exchange rate. They also found that behavioral complexity, not photorealism, drove the gain: crowd-like "Multi-group" scenes raised HOTA on DanceTrack from 68.8 to 73.5, while simple single-walker scenes hurt. We test whether the same holds when "behavior" means app usage context rather than human motion.

On the constraint side, Tran et al. benchmarked four learned traffic generators (a GAN, a diffusion model, a transformer and a state-space model). They found membership inference attacks up to 88% accurate and up to 100% recovery of network identifiers, with frequent identifiers memorized first. That rules out training a generative model on real plaintext traffic as a privacy-safe data source. It is especially dangerous here: Sivan et al. showed that leaked locations, though only 27% of places visited, covered 61% of users' dwell time. Homes and workplaces are exactly the "popular" values a sequence model would memorize.

Our design avoids this by never training on real users at all. That also means fidelity has to come from the real apps and their real third-party libraries rather than from a learned model.

Research Program

We would build a capture harness on Android emulators and physical test phones, plus a SmartThings test environment. Traffic is routed through a VPN-style capture client like the one Sivan et al. used, and device sensors are driven by injected values. A persona specification defines four things: a synthetic trajectory with stay points whose dwell distribution matches the concentration Sivan et al. report, a schedule of app interactions, a set of user-visible features exercised per session, and canary strings. The visible features include whether a map, weather widget or check-in screen was actually shown. The canary strings include unique pseudo-IMEIs, unique coordinates and unique hardcoded-status texts of the kind IoTWatch targets.

Each outbound plaintext flow is matched to canaries to establish exactly which sensitive values left the device and to which host. It is labeled legitimate if and only if the persona script had exposed a feature that needs that value at that moment. This is the "user intention" signal FlowIntent inferred heuristically from UI text, now known by construction. Payloads, headers, flow statistics and UI context are stored together. That lets the same corpus train lexical detectors, statistical-flow detectors and text-reasoning detectors.

The central experimental variable is persona diversity. At the low end, a persona follows one route with one app per session. At the high end, personas mix apps, alternate between visible and background use of the same feature, and hit dual-use hosts that serve both functional and advertising endpoints. The high-end conditions are the "alarm clock" cases where the same coordinate to the same domain is legitimate in one session and a leak in the next.

To check how close the generator is to reality, we borrow the method of Rosenberg-Kima and Pardos. They judged a model's realism by asking whether the gap between real data and its simulation exceeds the gap between two independent simulations. We apply that test to detector-relevant feature distributions, such as flow timing, destination mix and context-flow co-occurrence. The test tells us which features the synthetic users reproduce faithfully and which they do not, before any detector is trained.

The exchange rate is defined operationally. Let $F(n_r, n_s)$ be the F1 score of a detector trained on $n_r$ real and $n_s$ synthetic labeled flows and tested on held-out real-user traffic. Let $n_r^{\mathrm{full}}$ be the full real training set size. The equivalency ratio at real budget $n_r$ is

$$k(n_r) = \frac{n_s^{*}(n_r)}{n_r^{\mathrm{full}} - n_r},$$

where $n_s^{*}(n_r)$ is the smallest synthetic count with $F(n_r, n_s) \ge F(n_r^{\mathrm{full}}, 0) - \epsilon$. The substitutability is the largest fraction $1 - n_r/n_r^{\mathrm{full}}$ for which such an $n_s^{*}$ exists. We fix $\epsilon$ at one F1 point in advance.

Real test traffic comes from two sources. The first is the FlowIntent benchmark of 1,002 location-sharing instances and the SmartThings/IoTBench corpus from IoTWatch, if their authors release them; the file gives no public resource link for either. The second is a newly consented cohort captured with Sivan et al.'s protocol, which gives a test set no synthetic design choice can have influenced.

Alongside F1, we report a dwell-weighted recall: recall on illegitimate flows, weighted by the share of the user's time spent at the leaked location. This metric targets the home and work exposures that matter most.

Baselines

The naive reference point is a canary-free version of what practitioners already run. It matches regular expressions for coordinates and identifiers against plaintext payloads and flags any match sent to a known tracking domain. It needs no training data, so it is evaluated directly on the same real test sets. It sets the floor that any learned detector must clear on false alarms for legitimate flows. TaintDroid's study gives a sense of that problem: 37 of 105 flagged connections were legitimate (Enck et al.).

The strongest existing method is FlowIntent, which combines UI-derived intent features, 31 statistical flow features and URL tokens in a supervised classifier. It reports a 91.1% F-measure on its 1,002 real instances. We re-implement it and train it on 100% of the real training split. This "all real" model defines the target, and every synthetic-augmented variant uses the same architecture, the same feature set, the same hyperparameter search budget and the same held-out real test users. Beating it means one of two things. Either we match its F1 within one point using at most 40% of the real labels (a 60% replacement), or we exceed it when synthetic data is added to the full real set. For smart-home string leaks, the corresponding reference is IoTWatch's string classifier, which reports 94.25% accuracy and 95% precision. It is run the same way on the SmartThings split.

We add two competing synthetic sources, each trained under an identical labeling budget. The first is the learned-generator route. One transformer-style and one state-space-style generator, following the families benchmarked by Tran et al., are trained on the real training split. Detectors are then trained on their outputs, and both detector F1 and regurgitation of real coordinates and identifiers are measured. The second is the text-vignette route: the CI-RL model of Inan et al., trained on the Contextual Integrity Synthetic Dataset. It is applied zero-shot to a textual rendering of each flow and its UI context. CI-RL reports up to 40% leakage reduction on PrivacyLens, but there is no reported result on Lacuna for it on network flows.

The key-ingredient ablation holds corpus size fixed while dropping persona diversity to a single-route, single-app regime. A second ablation replaces scripted personas with random UI exploration and keeps canary labeling. Together these separate the contribution of behavioral structure from that of exact labels.

Why not just…?

Why not just train on the real labeled data you already have? This is the most serious threat. If FlowIntent-style detectors already saturate at a few hundred real flows, synthetic users add little accuracy. The equivalency curve settles it. A flat $F(n_r, 0)$ curve with no low-budget gap from synthetic augmentation would be a negative result, and we would report it as such. Even then, two benefits would remain. The synthetic corpus can be shared freely, which real leak traces cannot. It also covers labeled alarm-clock cases that real crawls produce rarely; FlowIntent needed 20,000 apps to yield 1,002 instances.

Why not just train a generative traffic model on real traces? Tran et al. found up to 88% membership-inference accuracy and full identifier recovery. Sivan et al. show that real plaintext leaks concentrate on homes and workplaces, the high-frequency values most likely to be memorized. Our learned-generator baseline measures this directly by counting how many real test-cohort coordinates and identifiers reappear in generated flows. A generator that also matches persona-based substitutability would still lose on that count.

Why not just apply an off-the-shelf contextual-integrity reasoning model zero-shot? Inan et al. trained on 729 text vignettes with keyword-matching rewards, which they acknowledge are fragile to reformulated payloads. Raw HTTP query strings and SDK parameter names are reformulated payloads in exactly that sense. The zero-shot CI-RL baseline tests this. If it rivals detectors trained on synthetic flows, the cheaper text route wins, and the proposal's value narrows to producing labeled evaluation data.

Why not just run real apps with random exploration and fake sensors, without personas? This is close to what we propose, so it is built in as an ablation rather than dismissed. Chang et al.'s DanceTrack result, where simple synthetic behavior degraded tracking, predicts that random exploration yields few context-contrasting pairs and transfers worse. If it does not, the persona machinery is unnecessary and the lighter method should be preferred.

Proposed Experiments

The first decisive check is the substitutability curve on real held-out users. It sweeps the real fraction over 0%, 20%, 40%, 60%, 80% and 100% and crosses it with synthetic volume on a logarithmic grid, for each synthetic source. It reports F1, false-alarm rate on legitimate flows and dwell-weighted recall. The proposal succeeds if diverse personas reach the all-real FlowIntent F1 within one point at 40% real data or less, while learned generators either fail to do so or regurgitate measurable real identifiers.

The second check is the diversity ablation at matched corpus size. Low-diversity personas should transfer measurably worse even when they generate more flows. If they do not, the behavioral-coverage hypothesis is wrong.

The third check is the realism test described above. It measures, per feature family, whether the real-versus-synthetic distance exceeds the distance between two synthetic seeds. We then check whether the features that fail this test are the ones whose removal hurts transfer least, which would show the diagnostic predicts transfer without retraining.

The fourth check is a leakage audit of every synthetic corpus. Exact and near matches of real-cohort coordinates, stay points and identifiers should be zero for persona data by construction. For the learned generators the count is unknown, and that count is the measurement.