Defending Machine Learning Models Against Extraction Attacks

807 concepts445 papers13 proposals

Training state-of-the-art machine learning models requires massive capital investments in data curation, computational infrastructure, and engineering expertise. To monetize these assets, providers expose their models via public application programming interfaces (APIs) under paradigms like Machine-Learning-as-a-Service (MLaaS) or Embedding-as-a-Service (EaaS).

However, this interface introduces a fundamental security vulnerability: model extraction attacks (or model stealing). By treating a proprietary API as a low-cost labeling oracle, an adversary can systematically query the system and use the outputs to train a local "student" model. This process replicates the functional utility of the target model at a fraction of the original development cost.

To secure intellectual property in an era of open API access, researchers have developed a multi-layered ecosystem of defenses. Understanding how these defenses work requires looking at model extraction not just as a data-theft problem, but as an optimization and information-theoretic challenge.

The Extraction Mechanism: Exploiting the API Oracle

To understand how to defend a model, we must first understand why extraction is so efficient. In a standard black-box setting, an attacker does not have access to the model's weights or training data. Instead, they leverage the continuous nature of neural network decision boundaries.

As demonstrated by Tribhuvanesh Orekondy et al. in Knockoff Nets: Stealing Functionality of Black-Box Models, an attacker does not even need data from the victim model's specific domain to steal its functionality. If a target model classifies specialized medical images, an attacker can query it using generic, out-of-distribution public images. The victim model will project these random inputs onto its learned manifold, returning confidence scores or high-dimensional embeddings. These outputs reveal the orientation, curvature, and margins of the model's decision boundaries.

When commercial systems rely on standardized, predictable architectures, the barrier to entry drops further. Manveer Singh Tamber et al. showed in Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models that proprietary embedding models can be replicated with high fidelity for under $300 by initializing a student model with a structurally similar open-source backbone. The student model absorbs the "dark knowledge" hidden within the teacher's soft probabilities or vector spaces, effectively executing unauthorized knowledge distillation.

Passive Defense: Watermarking and the Mechanics of Inheritance

If an adversary successfully extracts a model, how can the original owner prove intellectual property infringement in a court of law? The primary solution is post-hoc verification via watermarking or fingerprinting.

The core intuition behind watermarking is loss minimization alignment. When a student model trains on a watermarked teacher's outputs, its optimization objective forces it to minimize the discrepancy between its own predictions and the teacher's outputs. If the teacher systematically injects a subtle, hidden bias into its predictions, the student model will naturally inherit this bias as it converges.

For text generation models, Xuandong Zhao et al. developed a method in Protecting Language Generation Models via Invisible Watermarking that embeds a hidden sinusoidal signal directly into the output probability distributions during decoding. Because the student model is trained to mimic these exact distributions, the watermark is automatically transferred to the stolen clone.

[Attacker Query] ---> [Victim API (Injects Hidden Bias/Watermark)]
                             |
                             v
                     [Watermarked Output]
                             |
                             v
                     [Student Model Training] ---> [Stolen Clone Inherits Watermark]

However, sophisticated attackers can attempt to strip away watermarks by filtering outputs or fine-tuning the student model on a small set of clean data. To prevent this, defenses must move away from superficial output perturbations and toward structural entanglement.

In Entangled Watermarks as a Defense against Model Extraction, Christopher A. Choquette-Choo et al. introduced Entangled Watermarking Embeddings (EWE). Instead of treating the watermark as an independent task, EWE forces the model to use the same internal neural pathways for both the primary task and the watermark verification task. If an attacker attempts to optimize away the watermark behavior, the shared feature representations distort, causing a catastrophic drop in the stolen model's primary accuracy.

Alternatively, owners can use passive fingerprinting without modifying the model's outputs during deployment. Nils Lukas et al. demonstrated in Deep Neural Network Fingerprinting by Conferrable Adversarial Examples that one can generate a specific class of adversarial examples that exploit unique architectural quirks shared exclusively between a teacher and its distilled surrogates. This allows owners to identify stolen models with near-perfect accuracy while ignoring independently trained reference models that perform the same task.

Active Defense: Poisoning the Attacker's Gradients

While watermarking helps catch a thief after the fact, active defenses aim to disrupt the extraction process in real time. To understand why active defenses work, we must frame model stealing as an optimization problem where the attacker uses the API to compute empirical gradients.

When an attacker queries an API and receives a clean probability vector, that vector acts as a compass, showing the exact direction the student model's weights need to move to align with the teacher. If the defender can distort this compass, they can break the optimization process.

Bernt Schiele pioneered this approach in Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks. By introducing controlled perturbations to the API's confidence scores, prediction poisoning injects systematic noise into the attacker's loss landscape. This noise misdirects the student model's training gradients, causing its optimization path to stall or diverge.

Expanding on this concept, Mantas Mazeika et al. introduced Gradient Redirection in How to Steer Your Adversary: Targeted and Efficient Model Stealing Defenses with Gradient Redirection. Rather than adding random noise—which can degrade service quality for legitimate users—their method (GRAD$^2$) calculates the exact minimal perturbation required to actively maximize the error in the attacker's estimated gradients. The defense effectively "steers" the adversary's student model toward a useless or deceptive local minimum while maintaining high utility for benign queries.

Attacker's Goal:  [True Gradient Direction] ---------> [Optimal Clone]

GRAD^2 Defense:   [True Gradient] + [Perturbation] ---> [Deceptive Direction] ---> [Useless Model]

Proactive Defense: Designing Undistillable Models

The most robust defense strategy is to prevent knowledge transfer entirely by altering the model's information entropy. Knowledge distillation relies on the relative differences between the probabilities of incorrect classes. For instance, knowing that an image of a dog has a 0.1% chance of being a cat but only a 0.0001% chance of being a car tells the student model an immense amount about how the teacher structures its internal feature space.

If a model provider can withhold this structural information without changing the top-1 classification accuracy, the model becomes fundamentally harder to distill.

In Towards Undistillable Models by Minimizing Conditional Mutual Information, Linfeng Ye proposed a training framework called CMIM. By minimizing the conditional mutual information between the inputs and the soft output probabilities, the model is forced to collapse its non-dominant outputs into highly concentrated, uniform clusters. This strips away the "dark knowledge" hidden within the tail of the distribution. As a result, an adversary querying the API receives no more structural information than they would from hard, binary labels, neutralizing the efficiency advantage of distillation.

Similarly, Xinran Wang proposed Information Laundering for Model Privacy, which routes inputs and outputs through probabilistic filters. This mathematically bounds the amount of private model information an attacker can extract per query, turning model stealing into an economically non-viable strategy.

A Continuous Arms Race

Despite these advances, the boundary between defense and vulnerability remains fluid. As defenders build more sophisticated "nasty teachers" to block distillation, attackers design counter-measures.

For example, EN-HUI YANG et al. demonstrated in Markov Knowledge Distillation: Make Nasty Teachers Trained by Self-undermining Knowledge Distillation Fully Distillable that learned Markov transformations can map obfuscated teacher outputs back into a clean, distillable format. This ongoing tension ensures that model extraction defense remains one of the most active and critical frontiers in machine learning security.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 490 papers
  2. Version 1 · 2026-10-04 · 491 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_model_extraction_vulnerability_023353c2,
  title = {model extraction vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-extraction-vulnerability/rdr_023353c234f14788892b7d1480de8d98},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_model_extraction_vulnerability_023353c2_v2,
  title = {model extraction vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-extraction-vulnerability/rdr_023353c234f14788892b7d1480de8d98/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake