Machine extraction attacks clone a proprietary model by querying its API and training a copy on the returned outputs — which is, structurally, knowledge distillation with a teacher you do not own. Today's most capable attacks are either brute-force or blind: data-free methods perturb pixels with estimated gradients until the copy matches, burning millions of API calls, while prompt-evolution and active-learning methods poke at whatever data they happen to have. None of them treats the API call budget — the one resource that is scarce, metered, and monitored — as something to optimize directly.
This proposal treats query synthesis for model extraction as latent space optimization. We train a generative model on unlabeled, domain-adjacent data, map every possible query to a continuous latent code, and run Bayesian optimization there, where each candidate query is scored by how much new "dark knowledge" — the non-target structure in the teacher's soft probabilities — it would add to the student's training set. A probabilistic surrogate turns extraction from millions of blind probes into a few thousand deliberately chosen ones. Because the optimizer can be constrained to the generative manifold, every query also looks like ordinary in-distribution traffic, which neutralizes the growing family of defenses that only punish abnormal queries.
Core Claim
The claim is that model extraction should be formulated as sample-efficient distillation with an acquisition function over latent space, and that this simultaneously (a) cuts query budgets by one to two orders of magnitude relative to pixel-space data-free extraction, (b) evades distribution-based defenses by construction, and (c) exposes a hole in "undistillable model" defenses that were never tested against active query synthesis.
The gap is real and documented. Data-Free Model Extraction (DFME) formulates extraction as a generator-student game, but its generator searches raw pixel space with zeroth-order differences and needs 2M queries on SVHN and 20M on CIFAR-10 — a footprint any rate limiter catches. Stealix: Model Stealing via Prompt Evolution comes closest to our setting, but it evolves discrete text prompts with a genetic algorithm and inherits Stable Diffusion's prior, so it collapses on domains the generative backbone never saw. The machinery we want to import already exists on the benign side: InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models optimizes continuous soft prompts with Bayesian optimization and a kernel that couples latent distance to semantic distance, hitting 92% accuracy where discrete prompt search (Automatic Prompt Engineer) stalled at 4%; DIFER: Differentiable Automated Feature Engineering shows continuous search over decoded structures needs 40x fewer evaluations than discrete baselines. Nobody has pointed this machinery at the extraction problem, where the objective is not task accuracy but student-victim disagreement — the quantity that, per Knockoff Nets: Stealing Functionality of Black-Box Models, concentrates near decision boundaries.
Research Program
We build one system, LatentKnock, with four components. First, a generative query engine: an autoencoder (or diffusion latent encoder) trained on unlabeled data — no labels needed, so this stays "data-free" in the sense DFME and ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data use the term — that provides an encoder, decoder $G(z)$, and a density estimate over the latent prior. Second, the distillation core: the student $\mathcal{S}$ trains on all accumulated victim outputs using the $\ell_1$ loss on recovered logits, which DFME showed avoids the vanishing-gradient failure of KL divergence in black-box distillation. Third, the optimizer: a Gaussian Process surrogate over the disagreement function
$$d(z) = \big\|\mathcal{V}(G(z)) - \mathcal{S}(G(z))\big\|_1,$$
where $\mathcal{V}$ is the victim API and $\|\cdot\|_1$ is the elementwise sum of absolute differences over the output vector. Each BO round queries the argmax of an upper-confidence-bound acquisition
$$\alpha(z) = \mu_d(z) + \beta\,\sigma_d(z) - \lambda\,r(z),$$
where $\mu_d$ and $\sigma_d$ are the GP posterior mean and standard deviation of $d$, and $r(z)$ is a manifold penalty (negative latent-prior density) that keeps queries in-distribution. The GP kernel extends InstructZero's instruction-coupled kernel: latent distance is discounted when the victim's own outputs at the two points are dissimilar, so the surrogate's geometry tracks the victim's output geometry rather than raw latent distance — directly addressing the kernel-semantics mismatch that InstructZero identified as the failure mode of isotropic kernels.
Fourth, the defense stress test. Two published defenses make claims our attack is designed to check. Towards Undistillable Models by Minimizing Conditional Mutual Information (CMIM) collapses each class's output distribution to an invariant vector so that distilled students underperform label-smoothed baselines — but it was only evaluated against attackers training on fixed datasets, never against an adversary that searches for the boundary points where the collapse must be incomplete (a collapsed distribution still has to change somewhere across a class boundary). And Defense against Model Extraction Attack by Bayesian Active Watermarking (ACT), like BODAME: Bilevel Optimization for Defense Against Model Extraction, assumes boundary-seeking queries fall out-of-distribution, where the defender poisons or maximizes entropy. LatentKnock's queries stay on the in-distribution manifold by construction, so the trigger condition never fires.
Proposed Experiments
Victims are ResNet-50 classifiers (matching the CMIM and EuroSAT Benchmark setups, so reported numbers transfer) on CIFAR-10, SVHN, and EuroSAT — EuroSAT chosen deliberately because it is the domain where Stealix struggles (65.9% extracted accuracy against a 98.57% victim) and where public-data pools are thin. The headline metric is extraction accuracy (student test accuracy) at fixed query budgets of 10k, 50k, and 200k victim calls, plus queries-needed-to-match. DFME's reported target on CIFAR-10 is 88.1% (0.92x victim accuracy) at 20M queries; we count the proposal as successful if we reach within two points of that using at most 1% of its queries, and partially successful if only at 200k. On EuroSAT, beating Stealix's 65.9% by 15+ points at a comparable budget would show the latent prior transfers where diffusion-prompt evolution does not.
The defense experiments are the decisive checks. Against ACT, we measure whether the in-distribution constraint eliminates its reported 6–12% additional clone-accuracy reduction; against the CMIM teacher on CIFAR-100 (teacher accuracy 78.72%), we test whether latent boundary search yields a distilled student that exceeds the label-smoothing baseline — the specific criterion CMIM uses to define undistillability. Detection is measured with PRADA-style monitoring, where ActiveThief reports remaining undetected after 30,000 natural queries; we report the flag rate for our synthetic queries under the same detector, expecting that manifold-constrained latents pass for ordinary traffic. We also sweep the amount of unlabeled data available to the generative engine (1%, 10%, 100% of an unlabeled proxy set) to locate the point of prior failure honestly.
Baselines
The "do nothing" reference point is a student trained on victim outputs over randomly sampled latents from the same generative engine — same student architecture, same $\ell_1$ loss, same budget — which isolates whether the Bayesian optimizer, rather than the generative prior or the distillation recipe, does the work. The strongest existing attacker depends on the setting. For the query-efficiency question it is DFME, which reported 88.1% on CIFAR-10 at 20M queries and 95.2% (0.99x of the victim) on SVHN at 2M queries; we rerun DFME at our matched budgets (its authors' released recipe, tuned with the same grid we use for our own hyperparameters) and also report our method at its full budget, so the comparison holds in both directions. For the public-data regime it is ActiveThief, whose pool-based active selection achieved 61.52–98.18% agreement with victims across image and text tasks; we give it an unlabeled pool of the same size we use to train our generative engine, which is the fairest data allocation. On EuroSAT and the low-budget setting we include Stealix (65.9% EuroSAT; 73% on a HuggingFace NSFW filter at 500 queries per class). The key ablation is the kernel: replace the output-coupled kernel with an isotropic RBF kernel over latents, keeping everything else fixed. Beating the strongest baseline means, concretely: higher extraction accuracy at every matched budget, or equal accuracy at 100x fewer queries; evading ACT where DFME-style and unconstrained attackers do not; and, against CMIM, a student above the label-smoothing bar. If the isotropic-kernel ablation matches the full method, the contribution shrinks to "BO over a generative prior," and we will say so.
Why not just…?
Why not just select informative samples from public data with active learning? This is the most serious alternative, and ActiveThief is a genuine threat: its queries are natural, so it evades PRADA at 30,000 queries, and on CIFAR-10 with a large ImageNet pool it may well match us. We will run it head-to-head rather than argue. Our claim is that pool-based selection is bounded by pool coverage: it can rank existing points but cannot synthesize the boundary interiors a pool never sampled, which is exactly where DFME and Knockoff Nets showed the dark knowledge lives. If ActiveThief matches LatentKnock on the standard benchmarks, the contribution narrows to the regimes where it structurally cannot go — sparse-pool domains like EuroSAT, tiny budgets, and the CMIM defense test, where a pool drawn from the training distribution cannot contain the collapse-violating points our search targets. That narrowed claim is still worth having, and the experiments will tell us which world we are in.
Why not just do zeroth-order pixel-space search with more queries? DFME already does this, and it works — at 2M–20M queries. That budget is both detectable and, we argue, wasted: perturbing 3,072-dim pixels with forward differences spends most queries learning nothing, and the synthetic inputs it produces are exactly the out-of-distribution traffic that ACT and BODAME are built to poison. The matched-budget comparison settles the efficiency claim directly. If DFME at 200k queries matches LatentKnock at 200k, our efficiency claim fails; we consider that unlikely given that DFME's own results needed two orders of magnitude more queries, but it is the experiment that decides.
Why not just evolve discrete prompts genetically, as Stealix does? The evidence from the benign literature is consistent: InstructZero's continuous soft-prompt optimization reached 92% where discrete evolutionary search (APE) reached 4%, and DIFER's continuous latent search beat discrete search on 22 of 25 datasets with 40x fewer evaluations. Stealix's own numbers (49.6% on CIFAR-10) suggest the discrete bottleneck is real in extraction too. We still include Stealix as a baseline because it is the closest published attacker, and on EuroSAT — where its diffusion prior transfers — its 65.9% is the number to beat.
Why not just deploy better defenses instead of building a better attack? Because the defense literature has not been tested against this attacker. CMIM's undistillability claim was established only against fixed-dataset distillation, and ACT's watermarking only against out-of-distribution probing; both assumptions are exactly what latent optimization removes. Cheap alternatives are already ruled out by the file: BODAME reports that output rounding has almost no effect, and GRAD² notes that top-k truncation degrades calibrated confidence for legitimate users. A stress-testing attack is how we find out whether the expensive defenses hold.
Decisive Checks and Limitations
The proposal succeeds if the query-efficiency curve, the ACT evasion result, and the CMIM break all point the same way; it fails gracefully and informatively if only one does, since each component is independently interpretable. The honest limitation is prior dependence: LatentKnock needs some unlabeled data for its generative engine, so it is not data-free in DFME's strictest sense, and on domains absent from any obtainable proxy data — the PCAM/RetinaMNIST pathology case where Stealix's generative prior collapsed — our latent prior will inherit the same mismatch. We will report that boundary explicitly rather than hide it, and we will characterize how much unlabeled proxy data the engine needs before the attack degrades. The longer-term implication runs both ways: the same acquisition function, pointed at a model you own, becomes a tool for auditing where your own model leaks its dark knowledge before an adversary finds it.
References
- Data-Free Model Extraction
- Stealix: Model Stealing via Prompt Evolution
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models
- DIFER: Differentiable Automated Feature Engineering
- Knockoff Nets: Stealing Functionality of Black-Box Models
- ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data
- Towards Undistillable Models by Minimizing Conditional Mutual Information
- Defense against Model Extraction Attack by Bayesian Active Watermarking
- BODAME: Bilevel Optimization for Defense Against Model Extraction
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification