Reject, Don't Poison: Unsupervised Open-Set Gating Against Model Extraction

When you query a commercial prediction API, the model behind it was almost certainly trained under a closed-world assumption: it assumes every input belongs to one of the classes it saw in training. Model-extraction attackers exploit exactly this. Work on knockoff models showed that an adversary can steal a specialized classifier without any in-domain data, simply by feeding it random public images (say, generic ImageNet photos) and training a copy on the returned class probabilities, recovering 81% to 96% of the victim's performance (Knockoff Nets: Stealing Functionality of Black-Box Models). The closed-world softmax turns every out-of-domain query into free supervised training data. Current defenses either perturb the returned probabilities for every query — which is computationally expensive — or detect suspicious client accounts by comparing their query streams to normal traffic — which requires examples of attacks to tune the alarm threshold. This proposal argues for a third route: make the API itself open-set aware, so that it refuses to answer inputs outside its domain, and set the refusal and alarm thresholds with no attack data whatsoever, using only benign traffic. We then stress-test this gate against the most dangerous adaptive adversaries, including ones armed with classical frequency analysis.

Core Claim

The claim is threefold. First, out-of-domain rejection is the structurally correct response to extraction attacks that query with open-set data: it returns no posterior information at all, unlike output poisoning, and it requires no per-query gradient computation. Second, the detection threshold for "this account is extracting the model" can be set without any knowledge of the attack, by fitting a tail model (extreme value theory — a branch of statistics that models the distribution of the rare largest or smallest values) to benign account-level statistics, in the spirit of post-processing methods from operational open-set recognition. Third, the two obvious alternatives to outright rejection — truncating outputs and obfuscating labels — fail for reasons the literature already documents, and label obfuscation fails in a way we can make precise using Bayesian frequency inversion: any deterministic relabeling of outputs can be undone by matching observed label frequencies against public class statistics, the same attack that breaks substitution masking in encrypted databases (The Tao of Inference in Privacy-Protected Databases).

What Exists and What Is Missing

Four lines of work bracket this idea without covering it. Operational Open-Set Recognition and PostMax Refinement supplies the scoring machinery we reuse: it shows that in modern large vision backbones, unknown inputs produce higher feature magnitudes than known ones (the "feature magnitude surprise"), so dividing the maximum logit by the feature $L_2$ norm and fitting a Generalized Pareto Distribution (a standard tail model) on known validation data yields calibrated known-ness probabilities, reducing FPR95 by up to 28.1% relative to Maximum Softmax Probability (MSP) thresholding. But PostMax still needs surrogate out-of-distribution samples to lock an operational threshold, and it has never been applied to an adversarial API setting. Defense Against Model Stealing Based on Account-Aware Distribution Discrepancy (ADD) supplies the account-level view we adopt: per-query inspection fails because benign users with mild domain shift get flagged while curated attack queries look normal, so ADD instead compares each account's sliding window of query embeddings to class reference distributions via squared Fréchet distance (a distance between two probability distributions). But ADD sets its malicious-score threshold $\tau$ empirically, using simulated attack streams, and its own authors flag both the open problem of unsupervised threshold selection and an adaptive evasion in which the attacker blends benign in-distribution queries into the stream. Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks and How to Steer Your Adversary: Targeted and Efficient Model Stealing Defenses with Gradient Redirection represent the poisoning route and its cost: Jacobian computation or double backpropagation per query, which their successors try to amortize. ModelGuard: Information-Theoretic Defense Against Model Extraction Attacks even measures out-of-distribution detection as a preserved utility metric, but never uses it as a gatekeeper. Finally, The Tao of Inference in Privacy-Protected Databases proves that deterministic substitution mappings are invertible by maximum-likelihood frequency alignment solved with the Hungarian algorithm (an optimal assignment procedure), a result that has never been pointed at the label-obfuscation defenses in the extraction literature. No paper combines open-set rejection, unsupervised thresholding, and an adaptive frequency-analysis adversary; that intersection is the contribution.

Research Program

The system has three parts. The per-query score is PostMax's ratio, $s(x) = \max_k z_k(x) / \lVert f(x) \rVert_2$, where $z_k(x)$ are the unnormalized logits and $f(x)$ the penultimate features. The account-level statistic replaces raw per-query decisions with a window summary that is interpretable under benign traffic: for an account's recent query window $W$,

$$\hat{\theta}_W \;=\; \frac{1}{|W|}\sum_{x \in W} \mathbb{1}\!\left[\,s(x) < q_\beta\,\right],$$

the fraction of queries whose known-ness score falls below $q_\beta$, the $\beta$-quantile of benign scores estimated once from training data. Under benign traffic $\hat{\theta}_W$ concentrates near $\beta$; its tail is modeled with a Generalized Pareto fit on benign validation windows, and the alarm threshold $\tau$ is set as the quantile satisfying $\Pr_{\mathrm{benign}}(\hat{\theta}_W > \tau) = \alpha$ for a target false-alarm rate $\alpha$ (say 1%). This is the unsupervised threshold: no attack stream, no surrogate unknowns, and a guarantee that carries to whatever attack appears, because the guarantee is about benign traffic only. The response is rejection, not perturbation: once an account is flagged, its out-of-domain queries receive an explicit "unknown" response rather than a class posterior, cutting off the information channel that knockoff training depends on. Because the response is a cheap scalar comparison rather than per-query gradient work, it targets the latency overhead that motivated successors to MAD (How to Steal Your Adversary, which reported a 3.75x–6.33x speedup over MAD precisely because MAD's per-query cost was the bottleneck).

The adversarial analysis is where frequency analysis enters as a lens rather than a tool. We model two adaptive attackers. The blending attacker knows the account-level detector exists and mixes a fraction $\gamma$ of genuinely benign in-distribution queries into the stream, the exact evasion ADD flags but does not defend against; the analysis reduces to a contamination problem, with the detectable $\gamma$ determined by how far the attack's tail rate sits above $\beta$ relative to the noise in $\hat{\theta}_W$, and we will study robust variants of the statistic (trimmed means, median-based scores) to push the detectable $\gamma$ down. The obfuscation-inversion attacker targets any defense that answers suspicious queries with shuffled or relabeled outputs: treating the defender's deterministic relabeling as a substitution cipher, the attacker recovers the true mapping by Hungarian-algorithm alignment between observed label histograms and public class-frequency priors, exactly the machinery of The Tao of Inference. Demonstrating this inversion cleanly is what justifies rejection over obfuscation as the response primitive.

Baselines

The "do nothing" reference point is the undefended API, for which the literature already reports the cost: knockoff models recover 81%–96% of victim performance across tasks when trained on random public queries (Knockoff Nets), and the active query strategy of Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack reaches 60–95% of true model accuracy with only 1,000 to 8,500 queries. Any proposed defense must cut clone fidelity materially below these ranges to count. The cheapest real competitor is per-query MSP thresholding with a threshold tuned on benign validation data only; it is the natural first attempt at open-set gating, and PostMax's results — an FPR95 up to 28.1% worse than the norm-corrected score, driven by the feature magnitude surprise — predict it will misfire, but it must be run to show this on our protocol, on identical query logs and with the same tuning budget. The strongest competitor is ADD (Defense Against Model Stealing Based on Account-Aware Distribution Discrepancy), which reports detection AUROC of approximately 1.00 on CIFAR-10 and CIFAR-100, drops attacker clone accuracy from about 90% to near-random while causing 0% accuracy loss for legitimate users, and adds under 5ms of latency per query. It will be given every advantage: same query logs, same compute, and its threshold tuned on attack streams (which our method never sees). Beating ADD means three things: holding the achieved false-alarm rate near the 1% target when the test-time attack strategy differs from the one used for tuning, where we predict ADD's empirically tuned $\tau$ transfers poorly; surviving a higher blending fraction $\gamma$; and matching its latency. Finally, two ablations isolate the ingredients: the same gating pipeline with an empirically tuned threshold instead of the extreme-value-theory threshold (isolating unsupervised calibration), and per-query gating without the account-level window (isolating aggregation).

Why not just…?

Why not just return only the top-1 label? Prediction Poisoning explicitly evaluated passive defenses including rounding, top-$k$ truncation, and top-1 hard labels and found them ineffective against modern extraction, which reconstructs decision boundaries from coarse or top-1 information alone; How to Steal Your Adversary adds that truncation degrades transparency for benign users who need calibrated scores. Our experiments include a top-1-only condition so the comparison is direct rather than inherited.

Why not just add noise to, or shuffle, the outputs of suspicious accounts? Random jittering was shown to hurt the defender's own accuracy more than the attacker's, because noise averages out over a large query budget while the attacker's gradient direction survives (Prediction Poisoning), and ModelGuard showed uncalibrated noise falls to adaptive attackers who model the noise distribution. The more subtle case is deterministic label shuffling, which sounds safer than noise; Experiment 4 below is designed to break it, applying the Hungarian-alignment frequency attack of The Tao of Inference to a shuffling defense. If, as we expect, the attacker recovers the mapping and most of the clone fidelity, this is direct evidence that obfuscation is the wrong primitive and rejection is the right one. We are honest that this is a prediction: if shuffling survives frequency inversion at realistic query budgets, the rejection-only story weakens and the proposal would pivot to combining rejection with the quantization ideas of ModelGuard.

Why not just run a per-query out-of-distribution filter at the gateway? This is the simplest version of our own idea, and the literature documents its failure mode twice over. ADD's authors showed per-query inspection flags benign users with mild domain shift (USPS digits against an MNIST model) while letting curated attack queries through, which is why they moved to account level; and PostMax showed that on modern backbones, naive per-query confidence thresholds fail badly (FPR95 up to 28.1% worse) because unknown inputs inflate feature magnitudes. The per-query ablation in our baseline set makes this quantitative on the extraction protocol rather than relying on the transfer of results across settings.

Why not just deploy ADD and tune its threshold harder? This is the most serious objection, because ADD's reported detection numbers (AUROC ≈ 1.00) leave little headroom on the attacks it was tuned for. Our claim is not better AUROC on seen attacks; it is calibration robustness and adaptivity. Experiment 1 is built to settle this: if ADD's threshold, tuned on one attack family, holds its false-alarm rate and detection on an unseen family (Marich-style active sampling versus Knockoff-style random sampling, and the blending attacker), then the unsupervised-threshold contribution collapses and the proposal reduces to "ADD plus rejection," which we would report as such.

Proposed Experiments and Decisive Checks

Experiment 1 tests threshold transfer. Victims are trained on CIFAR-10, CIFAR-100, and MNIST (the ADD/MAD setting) plus a CUB-200 victim queried with ImageNet and OpenImages images (the Knockoff Nets protocol) and a BERT victim trained on BBC News and queried with AGNews text (the Marich protocol). ADD is tuned on random-query attacks and tested under active sampling, and vice versa; our gate is calibrated once from benign traffic at $\alpha = 1\%$ and tested under all attacks. Metrics are achieved false-alarm rate on benign accounts, detection AUROC, and, critically, the false-rejection rate for benign shifted users (USPS against MNIST, Indoor67 and SVHN against Caltech256/CIFAR victims, following the exposure sets of ModelGuard). Experiment 2 measures end-to-end theft: clone fidelity under no defense, MSP gating, our gate, ADD (which poisons on trigger), and MAD-style poisoning, at matched benign utility. Experiment 3 sweeps the blending fraction $\gamma$ and reports the minimum detectable $\gamma$ for ADD, our base statistic, and its robust variants. Experiment 4 runs the frequency inversion: a defender that shuffles labels for flagged accounts, attacked by Hungarian alignment against public class priors; the metric is clone fidelity relative to the undefended baseline, where near-full recovery would vindicate rejection. Experiment 5 measures per-query latency against ADD's reported sub-5ms overhead and MAD's Jacobian cost; the target is to stay in the scalar-comparison regime. The decisive checks are two: if ADD's empirically tuned threshold transfers across attack families within a factor of two of its target false-alarm rate, the unsupervised-calibration claim fails; and if a blending attacker with $\gamma \ge 0.5$ defeats both the base and robust statistics while ADD also falls, the account-level paradigm itself needs revisiting, and we would say so. No reported result found on Lacuna for the blending-attacker detection rates of any method, which is precisely why Experiment 3 is worth running.