Do Synthetic Fraud Benchmarks Rank Detectors the Way Real Data Does?

Fraud and money-laundering research has a reproducibility problem. The data that matters, such as bank transactions or platform abuse logs, cannot be shared, so the field increasingly relies on synthetic benchmarks: simulated transaction worlds, rule-generated fraud graphs, and fraud cases injected into real records. A benchmark is only useful if it gets one thing right. When method A beats method B on the synthetic data, A should also beat B on real data. No one has checked this. This note proposes a controlled audit of that property, which we call rank fidelity, together with a cheap diagnostic that predicts when it fails. The diagnostic is a deliberately trivial detector. If simple statistics of the simulator's output, such as node degree, transaction amounts, or timing, already separate fraud from benign activity, the benchmark is probably rewarding simulator artifacts and not fraud structure.

The core experiment can be run with public data. We take public fraud graphs that carry real labels, throw away the real fraud, inject synthetic fraud into the real benign background using the generation recipes the field actually uses, and ask whether a fixed set of detectors is ranked the same way on the injected labels as on the real ones. Because the background is identical, any change in ranking comes from how the fraud was synthesized and not from a change of domain. The proposal then tests one specific fix. Injected fraud is constrained to match benign nodes on the trivial statistics, much as real fraudsters camouflage themselves to look like legitimate shipments. The question is whether this matching restores rank fidelity.

Core Claim

We claim that common synthetic fraud injection produces benchmarks that are reproducible but measure the wrong thing, in the same way synthetic safety benchmarks did for language models. The Illusion of Intent showed that linear probes reaching 1.0 AUROC on InjecAgent and AdvBench fall below 0.10 once the top keywords are removed. In other words, a clean and repeatable number was tracking surface vocabulary. We expect the graph analogue to hold. Synthetic fraud is separable by low-order signatures that real fraud lacks. As a result, rankings on synthetic data correlate weakly with real-label rankings, and how well a trivial-feature detector scores on a synthetic benchmark predicts how poorly that benchmark's rankings transfer. The constructive half of the claim is that matching injected fraud to benign nodes on those signatures raises rank fidelity. If it does not, the failure lies in higher-order structure, and that result is itself informative.

Relation to Prior Work

Three papers motivate this work by building synthetic benchmarks precisely because real data is locked away. Each leaves transfer unmeasured. Realistic Synthetic Financial Transactions for Anti-Money Laundering Models releases AMLworld, an agent-based laundering simulator. Its authors argue synthetic labels can beat real ones because they contain no undetected crimes, but they never test models trained on AMLworld against bank data. Association-Focused Path Aggregation for Graph Fraud Detection builds G-Internet from 12 fraud rules and acknowledges its simulated nature. Evaluating Fairness in Transaction Fraud Models draws fairness conclusions from the Sparkov and IBMCard simulators without checking real populations. Our work does not build another simulator. It measures whether this style of benchmark preserves orderings of detectors.

Two papers use the injection practice we audit directly. Detecting Suspicious Timber Trades evaluates on synthetic perturbations of real customs records. Federated Continual Learning to Detect Accounting Anomalies reports 99.84% precision against anomalies injected into municipal payments. Both concede that real camouflaged fraud may behave differently. Our controlled setting turns that concession into a measurement.

Benchmarking Fraud Detectors on Private Graph Data both constrains and supplies parts of the design. It shows that private scoring servers leak membership through returned accuracy, which it calls "Accuracy-as-an-Oracle." It also shows that differentially private synthetic graphs (stochastic block models and related generators) need so much noise that they cannot separate good detectors from bad ones on Amazon and YelpChi. We reuse those generators as a baseline arm. We also take the negative result seriously: our fix keeps the real benign graph and changes only how fraud is placed, which is a different trade-off from privatizing the whole graph.

The diagnostic borrows its logic from lexical ablation in The Illusion of Intent. It is also motivated by evidence that real adversaries defeat rule-based synthesis. Hiding in Plain Sight shows this for homoglyph-masked hate speech, and Negative Deceptive Opinion Spam shows it for crowdsourced deception. Neither of those works studies graphs or rankings.

Research Program

The testbed consists of five public graphs that carry real fraud labels: YelpChi, Amazon, T-Finance, T-Social, and Elliptic. These are real but anonymized, so they serve as stand-ins for private production data.

For each graph we keep the full structure and node features. We remove the real fraud labels and generate synthetic fraud under several recipes:

  • random anomaly injection, the common default;
  • rule-template injection modeled on the G-Internet rule style, including multi-hop chains through benign bridge nodes;
  • context-preserving perturbation of benign nodes in the style of MEAD;
  • the differentially private generators from the private-benchmarking paper.

A fixed set of detectors is then trained and evaluated on each version: logistic regression and LightGBM on node features, GCN, GAT, CARE-GNN, GDN, PNA, and GPA from the official GPA code. Each detector gets the same tuning budget on every version.

Rank fidelity is the Kendall rank correlation between detector orderings by average precision (AP) on the synthetic labels and on the real held-out labels. We also record whether the top-ranked detector agrees, and the signed error in each detector's estimated improvement over GCN.

The diagnostic is a "signature detector": gradient boosting restricted to degree, local clustering, amount and time statistics, and counts of edge types. Its AP on a synthetic benchmark measures how much of the task can be solved by surface statistics. We will test whether this AP, computed only on the synthetic data, predicts rank fidelity across recipes and graphs. If it does, it can be published alongside any synthetic benchmark at no privacy cost.

The fix, signature-matched injection, chooses injection sites and rewires injected edges so that the joint distribution of the signature features is the same for fraud and benign nodes. We use propensity-style resampling and degree-preserving rewiring. The intended effect is that only higher-order patterns remain to distinguish fraud.

A secondary, explicitly confounded analysis ranks the same detectors on AMLworld against T-Finance and Elliptic, and on G-Internet against T-Social. The domains differ, so this is a sanity check on the existing benchmarks and not a clean test.

Baselines

The naive reference point is random anomaly injection evaluated by plain AP, which is how most injected-anomaly papers report results. Matching or exceeding its rank fidelity is the minimum bar.

The strongest existing competitor is rule-template injection in the G-Internet style. It is the most deliberate attempt in the file to encode real fraud mechanics, and its authors report GPA gaining "+15.7% improvement in Average Precision" on G-Internet and "+9.2%" on T-Social over mainstream detectors. If rule templates already preserve rankings, our fix is unnecessary. We run it on the same real backgrounds, with the same number of injected nodes and the same detector tuning budget.

The second strong competitor is the differentially private synthetic graph route of Goldberg et al., run with their generators at matched privacy budgets. That paper reports qualitatively that these generators fail to separate detectors, but I found no reported rank-correlation number for any of these methods on Lacuna. The only related ranking-type evidence is AMLworld's isolated-bank versus shared-graph contrast, "average F1 score of ~8.7%" versus "22.1%" on LI-Large, which concerns data access and not fidelity.

The ablation that isolates our key ingredient matches only degree, or only amount and time statistics, rather than the full signature set. This shows which signatures carry the artifact.

Beating the baselines means higher Kendall correlation with real-label rankings on at least four of five graphs, with confidence intervals from detector-seed bootstraps that exclude the rule-template result. It also requires that the signature detector's AP on our matched benchmark be near chance while the real-label ranking stays recoverable.

Why not just…?

Why not just benchmark on the public real graphs and drop synthetic data? This is a real threat, and the audit will report it if it holds. However, five anonymized graphs cannot cover laundering cycles, fairness audits, or domains with no public labels, which is why AMLworld, Sparkov and G-Internet exist. The useful question is therefore which synthetic recipes can be trusted where no real data exists. The five real graphs are what let us answer it.

Why not just measure distributional similarity between synthetic and real graphs? Degree-distribution or embedding distances are cheaper. But a benchmark can match the overall distribution while the fraud class remains trivially separable, and that is the keyword failure in The Illusion of Intent carried over to graphs. We will include standard similarity scores as competing predictors of rank fidelity. If they predict it as well as the signature-detector AP, we will recommend them instead.

Why not just use differential privacy on the real graph? Benchmarking Fraud Detectors on Private Graph Data already finds that the noise required erases the signal that ranks detectors, and its private-scoring alternative leaks membership. We rerun its generators as a baseline arm in case our rank metric is more forgiving than its utility measures.

Why not just use richer rule templates? More rules may help. The rule-template arm and the ablations test whether added mechanistic detail or signature matching is what moves rank fidelity. Rules and matching can also be combined, and we will report that combination.

Decisive Checks

The proposal fails if random or rule-based injection already gives Kendall correlation with real-label rankings comparable to signature-matched injection, or if signature-detector AP does not predict rank fidelity across recipes. Two kinds of temporal check probe whether the measured fidelity is stable. On Elliptic, we repeat the audit across time splits to see whether fidelity measured on early periods holds under later drift. Separately, we verify that the matched benchmark's rankings are stable across injection seeds, since an unstable benchmark would trade one reproducibility failure for another.