Stealing the Gate: Abstention Leakage in Malware Triage APIs

Malware inspection at enterprise scale is a two-tier system. A fast static model, reading only the file's bytes and headers, answers most inputs in milliseconds; the uncertain remainder is deferred to a dynamic sandbox that actually executes the file and watches what it does. This is selective prediction in disguise — a model allowed to say "I won't answer this one, send it to the expensive tier" — and it works remarkably well: on the EMBER2018 corpus of 800,000 Windows executables, Nguyen et al. showed that an uncertainty-gated static model matched the accuracy of running the expensive tier on everything, while invoking it on only 13.2% of test samples Minimizing Compute Costs: When Should We Run More Expensive Malware Analysis?. But the gate that decides who gets scrutinized is itself exposed through the same public API, and every response it returns carries an implicit bit: this file was — or was not — worth slow inspection. That bit, plus the response latency (sandboxing is up to 26,000× slower than static analysis), is an oracle an adversary can interrogate.

This note proposes treating the abstention decision as a leakable secret. The claim is that an adversary can extract the deferral boundary of a triage API with cheap synthetic queries, and that this extracted boundary is worth more to an adversary than the predictor itself — because it tells an evasive payload exactly where to sit so it receives a fast, confident "benign" answer and never detonates in the sandbox. The proposal formalizes this channel, demonstrates the combined extraction-plus-evasion attack, and builds a deferral policy that is both evasion-aware (it predicts whether the sandbox will be blinded before routing there) and extraction-resistant (its boundary is keyed and non-stationary, so stolen maps go stale).

Core Claim

Model extraction — training a copy of a victim model by querying its API — is usually framed as stealing the predictor $f(x)$. Existing defenses accordingly perturb what the API says: Orekondy et al. rotate the attacker's gradient updates within a utility envelope (MAD) Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks, Wu et al. multiply posteriors by a learned noise transition matrix (EMMA) Efficient Model Stealing Defense with Noise Transition Matrix, and Allouah et al. mix teacher logits with a proxy student's (PoE) The Distillation Game: Adaptive Attacks & Efficient Defenses. All of these assume the API must answer every query; none touches the abstain/answer decision, which survives even top-1 hard-label truncation. We claim three things. First, the selection function $g(x)$ of a hierarchical API is an extraction target in its own right: one bit per query, at a large budget, reconstructs its boundary, and latency makes the bit observable even if the API tries to hide it. Second, leaking $g$ is strategically worse than leaking $f$ in inspection pipelines, because the deferred tier is not a reliable oracle — environment-aware evasion, where malware detects virtualization and suppresses its payload, is exactly the mechanism Dambra et al. show collapses dynamic classification once more than 65% of a family's samples fail to detonate Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance. An adversary who knows where the gate's defer region ends needs only confident-looking statics plus standard anti-VM routines Examiner: Automatically Locating Inconsistent Instructions Between Real Devices and CPU Emulators for ARM. Third — and this is the twist — the "obvious" fix of having the defender abstain on suspicious-looking queries is self-defeating: an abstention pattern is itself a labeled map of the boundary's most informative regions, handing the adaptive attacker exactly the signal Allouah et al. showed they can exploit The Distillation Game: Adaptive Attacks & Efficient Defenses. Abstention is simultaneously a defense and a leak, and the design problem is to keep the first while closing the second.

The defense we build has two ingredients. The gate keeps its internal confidence signal private and scale-invariant: following Selective Classification Under Distribution Shifts, which showed softmax confidence collapses under shift while raw-logit margins do not, the gate scores inputs by $m(x)$, the difference between the two largest logits. The public decision is a keyed, non-stationary threshold:

$$\tilde{g}_t(x) \;=\; \mathbf{1}\{\, m(x) \;\ge\; \gamma_0 + \epsilon_t \,\}$$

where $\gamma_0$ is the published operating point and $\epsilon_t$ is a secret offset redrawn on each key epoch. Honest users experience a coverage that fluctuates mildly around the operating point, while any boundary map the attacker extracted during epoch $t$ has irreducible error on the order of $|\epsilon_t|$ thereafter. The second ingredient is a detonation head $\hat{s}(x)$, trained on sandbox logs to predict the probability that dynamic analysis will actually yield informative behavior; uncertain samples with low $\hat{s}(x)$ are routed to deep static disassembly (CAPA-style) rather than to a sandbox the payload will simply blind. This directly answers the gap Nguyen et al. leave open — they defer on entropy but never ask whether the expensive tier will be evaded Minimizing Compute Costs: When Should We Run More Expensive Malware Analysis?.

Research Program

Three workstreams. The first characterizes the channel: an information-theoretic and empirical analysis of how accurately the deferral boundary can be estimated from $N$ one-bit (plus latency) observations, for both passive uniform probes and active probes that reweight by a one-step gradient-alignment value function, the adaptive strategy Allouah et al. used to lift student accuracy against a strong defense from 34.3% to 51.5% on GSM8K The Distillation Game: Adaptive Attacks & Efficient Defenses. The second builds the attack end-to-end: knockoff-style query synthesis — Orekondy et al. showed random open-dataset queries project onto the victim's manifold and expose boundary orientation Knockoff Nets: Stealing Functionality of Black-Box Models, and even commercial embedding models fall for under $300 Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models — followed by a payload-placement optimizer that pushes a genuinely malicious binary into the region where the gate answers (never defers) with high benign confidence, wrapping it in anti-VM triggers as insurance. A dual variant floods the extracted defer region with crafted files to exhaust sandbox compute. The third builds and stress-tests the defense, including the honest possibility that keyed thresholds and decoy abstentions trade coverage for secrecy at rates we must measure rather than assume.

Proposed Experiments

The static tier and gate are built on EMBER2018 (800k Windows PE files, 600k/200k temporal split), replicating the Nguyen et al. setup: MalConv plus a LightGBM static model, with the original EMBER paper's LightGBM baseline (AUC 0.999; 93% detection at 0.1% false-positive rate) as a sanity anchor EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. The dynamic tier uses Speakeasy emulator traces with Nebula, the transformer that reaches 0.570 true-positive rate at a $10^{-3}$ false-positive rate on incomplete behavioral logs Nebula: Self-Attention for Dynamic Malware Analysis, and the detonation head is trained on the JuanLesPIN sets, where Dambra et al. provide family-level feature-missing rates and out-of-distribution families (static F1 0.957 versus dynamic 0.926 in-distribution; static accuracy falling from about 0.95 to 0.81 on unseen families) Decoding the Secrets of Machine Learning in Malware Classification. Four metrics carry the paper: boundary-recovery error (the attacker's estimated boundary versus the true one, on held-out probes, as a function of query budget); evasion success (fraction of malicious samples answered "benign" without detonation); forced-deferral rate (the denial-of-service variant); and honest-user risk–coverage curves plus latency overhead for the defense. Every defense number is reported under an adaptive attacker who knows the defense family and re-optimizes — the lesson of The Distillation Game is that passive evaluation flatters defenses. The decisive negative control is a bound on the channel itself: if the abstain bit proves too uninformative to reconstruct the boundary at realistic budgets, the extraction half of the claim fails and we will report that, leaving the evasion-aware routing contribution to stand on the FMR evidence alone.

Baselines

The naive reference point is an always-answer static-only API with no gate at all — MalConv/LightGBM answering every file in milliseconds. It is the "do nothing" case that isolates what the gate and its leakage are worth: against it we measure how much evasion the adversary achieves without any boundary knowledge, and how much honest accuracy is lost by never consulting dynamic analysis (per Dambra et al., roughly the 0.95-to-0.81 out-of-distribution drop). The strongest existing competitor is the Nguyen et al. learning-to-defer gate: a Bayesian MalConv with Monte Carlo dropout whose predictive entropy triggers rejection or deferral, which on EMBER2018 matched full-ensemble accuracy while requiring dynamic analysis for only 13.2% of test samples, and predicted CAPA capability outputs with 98% accuracy Minimizing Compute Costs. We will replicate it on the identical EMBER2018 split with the same static model, the same dynamic budget, and the same tuning protocol, so any difference is attributable to the detonation head and keyed threshold, not to compute. Beating it means three concrete things at once: a lower missed-malicious rate among answered samples on the JuanLesPIN out-of-distribution and high-FMR families at comparable deferral budget; materially higher boundary-recovery error under the adaptive attacker; and latency overhead staying near 1× (both added heads are single forward passes, versus the 2.93× generation overhead of antidistillation sampling and 1.64× for PoE reported on GSM8K The Distillation Game). A second strong baseline is EMMA ported to this pipeline — noise-transition-matrix posterior perturbation with no abstention change, which on CIFAR-100-to-CIFAR-10 extraction reduced attacker accuracy by 23% at a 1% defender cost Efficient Model Stealing Defense with Noise Transition Matrix; running it here isolates whether posterior perturbation alone can close a channel that lives in the abstain bit, which we predict it cannot. Finally, two ablations carve out the proposal's ingredients: the full gate with the detonation head removed (does routing by predicted detonation failure matter, or is plain entropy deferral enough on evasive families?), and the full gate with the threshold key frozen (does non-stationarity actually degrade extracted maps, or is one clean extraction enough forever?).

Why not just…?

Why not just return hard labels or round the probabilities? This is the first thing an API operator reaches for, and it does nothing here: Orekondy et al. showed attackers reconstruct high-fidelity clones from top-1 labels alone Prediction Poisoning, and more fundamentally our channel is not the posterior — it is the abstain/answer bit and the latency gap, both of which persist untouched under any amount of output truncation.

Why not just add random noise to outputs? Refuted twice in the literature: randomly jittering outputs hurts the defender's accuracy more than the attacker's Prediction Poisoning, and students filter noise across large query budgets Efficient Model Stealing Defense with Noise Transition Matrix. Again, none of it touches the abstain bit.

Why not just use a standard out-of-distribution detector to reject exploratory queries? Feng et al. showed softmax-based OOD scores collapse under the covariate and label shift that synthetic extraction probes deliberately induce, that margin scores (which we adopt) dominate them across ImageNet, CIFAR, and iWildCam, and that AUROC-style thresholds ignore the accuracy of what is actually answered Selective Classification Under Distribution Shifts. Beyond scoring, a static reject rule is a stationary boundary — precisely what the attacker is extracting.

Why not just run the sandbox on everything and skip the gate? Compute makes it impossible: dynamic analysis is up to 26,000× slower than static, and a 20-million-file production batch would need roughly 333 years of sandbox time; deferral matched the ensemble while sandboxing 13.2% of samples Minimizing Compute Costs. The gate exists for an economic reason, which is exactly why it is worth attacking and defending.

Why not just drop dynamic analysis and go static-only, since sandboxes are evadable? Static models generalize poorly to unseen malware families — accuracy dropping from about 0.95 to 0.81 — while detonated dynamic behavior generalizes better Decoding the Secrets. Static-only inspection concedes the novel-family regime to the adversary; our detonation head is designed to keep the dynamic tier useful exactly where it is currently wasted.

Why not just hide the abstention — answer every query and detonate silently in the background? This is the most serious alternative, and we treat it as a real threat rather than arguing it away. It closes the explicit bit but not the timing channel: a 26,000× latency difference is observable, and adversaries profiling analysis engines through timing anomalies is established practice PanoptiChrome: A Modern In-browser Taint Analysis Framework. Our experiments include a latency-masking variant (constant response time, asynchronous detonation) to measure how much of the channel survives; if masking plus queuing closes most of it at acceptable user latency, that will be an honest competing finding, and the keyed threshold remains as defense-in-depth for deployments that cannot afford to fake sandbox latency on every request.

Why not just deploy PoE or EMMA on the gateway? Both are strong against posterior-reading students, and EMMA composes with our gate as an orthogonal layer. But both answer every query and leave the deferral decision untouched, and the adaptive-attack evidence — a strong defense's margin collapsing from 34.3% to 51.5% student accuracy once the attacker filters by value The Distillation Game — warns that anything stationary, including a fixed perturbation, is eventually mapped. Our contribution is the piece they structurally cannot provide: making the decision to answer itself hard to pin down.

Decisive Checks

Three findings would settle the proposal. A boundary-recovery curve showing the adaptive attacker reconstructing the undefended gate's defer region at practical query budgets, while its error stays pinned near the key scale $|\epsilon_t|$ against the defended gate, establishes the leak and the fix. An evasion-success gap between adversaries with and without boundary knowledge — on JuanLesPIN out-of-distribution families where static confidence is already weakest — establishes that stealing the gate, not the predictor, is what pays. And an honest risk–coverage curve for the jittered, detonation-aware gate within a few points of the Nguyen et al. replication at a comparable 13–17% deferral budget establishes that secrecy was bought at a price a production pipeline would actually pay. If any leg fails, the negative result still maps the terrain: it would tell API operators whether the thing worth protecting is what their model says, or merely whether it chose to speak.