Antidistillation is Cheap: Lightweight Distillation-Resistance via Stylistic Corruption

Anh Tu Ngo, Anupam ChattopadhyayarXiv 2026 · arXiv

TL;DR

This paper introduces a lightweight defense against unauthorized model cloning by injecting "style-steering" vectors into a teacher LLM's activations during inference. By dynamically corrupting the stylistic signal of responses served to suspected adversaries, the method successfully disrupts the knowledge distillation process for student models while maintaining semantic accuracy and incurring near-zero computational overhead.

What Problem Does This Paper Address?

Building frontier Large Language Models (LLMs) is incredibly expensive. However, competitors can "cheaply" replicate these capabilities through Knowledge Distillation (KD): querying a powerful "teacher" model via an API and using its responses to fine-tune a smaller, cheaper "student" model.

Existing defenses against this intellectual property theft are often impractical. Some rely on "watermarking" (which only detects theft after the fact), while others use "anti-distillation sampling" or "proxy models" that significantly increase inference latency and memory usage. The authors aim to create a defense that is proactive (prevents learning), efficient (no latency hit), and natural (the output shouldn't look like gibberish to the attacker).

Methods

The authors propose a framework that selectively activates when a user is flagged as malicious (e.g., via query volume or distribution).

Figure 1: Overview of our lightweight distillation-resistant defense framework. For benign queries, the teacher model T generates standard responses. Upon detecting malicious queries, the framework dynamically switches the output style by injecting pre-computed style-steering vectors into the hidden activations at inference time, disrupting student model distillation while maintaining semantic coherence.

The core technique is Stylistic Corruption via Contrastive Activation Addition (CAA):

  1. Sourcing Styles: They curate datasets of contrastive pairs: a "base" response and a "styled" response (e.g., using a "Scratchpad" or "Jargon" persona) for the same prompt.
  2. Steering Vectors: They calculate the difference in mean activations between these pairs across the model's hidden layers to create "steering vectors."
  3. Inference-Time Injection: When a malicious query is detected, these vectors are added to the model's residual streams. This shifts the output style without changing the underlying weights or requiring a second "proxy" model.
  4. Dynamic Switching: To prevent adversaries from reverse-engineering the defense, the system randomly rotates between four personas—Scratchpad, Distractor, Dead-end, and Jargon—on a per-query basis.

Datasets

The researchers evaluated their defense across several standard benchmarks:

  • Open-ended Reasoning: GSM8K (grade school math) and MATH500 (challenging competition math).
  • Multiple-Choice: AI2 Reasoning Challenge (ARC) and CommonsenseQA (CSQA).
  • Steering Data: They sampled 100 problems each from GSM8K and ARC to generate the contrastive pairs needed to build the steering vectors.

Key Insights and Ideas

  • Style is the Signal: Recent research suggests that student models gain reasoning capabilities largely by imitating the style of the teacher's reasoning traces. By corrupting this stylistic signal, the "dark knowledge" the student tries to extract becomes inconsistent and harder to learn.
  • Covert Traps: Unlike blunt rate-limiting or bans, this defense acts as a "covert trap." The adversary continues to receive semantically correct answers, leading them to spend financial resources harvesting data that they only realize is "un-distillable" after they've already trained their student model.
  • Activation Engineering vs. Sampling: By intervening at the activation level rather than the token-sampling level, the defense avoids the "unnatural artifacts" (like nonsensical tokens) often produced by previous anti-distillation methods.

Key Results

The defense was tested using Gemma-3-12b-it as the teacher and several models (Qwen, Phi, Llama) as students.

  • Distillation Disruption: On GSM8K, student models trained on "steered" data saw accuracy drops ranging from 9% to over 11% compared to those trained on pristine data.
  • Teacher Utility: The teacher model's own accuracy remained largely intact, with only a negligible drop (e.g., ~2.8% on GSM8K), ensuring that the "trap" remains convincing.
  • Efficiency: As shown in Figure 12 (not provided here, but discussed in text), the method introduces zero latency overhead, maintaining the same tokens-per-second throughput as an undefended model.
  • Comparison: The authors found their method achieved comparable anti-distillation effects to state-of-the-art methods like ADS and DOGe but with significantly lower development time and inference costs.
Figure 2: Evaluation results on GSM8K and MATH500 showing base performance and percentage decrement under style steering (Teacher) or sequence-level distillation (Student models).

Limitations and Future Work

  • Detection Dependency: The framework assumes the existence of a reliable "malicious query detector." If the detector fails (false negatives), the adversary gets clean data; if it over-triggers (false positives), benign users get weirdly styled responses.
  • Adaptive Attackers: The authors admit that a "strong LLM normalizer" (an attacker using another LLM to rewrite and "clean" the traces before training) can partially undermine the defense. Strengthening the coupling between stylistic corruption and task-relevant content is a suggested path forward.
  • Domain Complexity: While effective for math and logic, the impact of style steering on more creative or subjective domains remains an open question.

Summary updated

Abstract

Knowledge distillation (KD) is a highly effective paradigm for transferring capabilities from large language models (LLMs) to smaller, more specialized student models. In commercial deployment where frontier models are restricted behind black-box APIs, sequence-level distillation remains a potent vector for competitors to clone proprietary capabilities at a fraction of the development cost. Existing defenses mostly rely on surrogate models to manipulate next-token sampling, which increases computational overhead and real-time inference latency. In this work, we propose a lightweight defense that dynamically injects style-steering vectors into the teacher model's activations upon malicious queries. This framework offers exceptional flexibility, allowing hosts to customize target defensive styles and dynamically blend them to maximize stylistic variance. Crucially, this inference-time intervention preserves semantic utility and fluency during malicious queries, misleading adversaries into believing that they are extracting pristine teacher outputs, while standard consumers continue to receive original, unmodified responses. Our experiments demonstrate that this framework successfully disrupts student learning, achieving substantial anti-distillation performance while incurring negligible overhead and preserving text naturalness. Our code is available at https://anonymous.4open.science/r/distill-resist-38C6/.
BibTeX
@misc{ngo2026antidistillation,
  author = {Anh Tu Ngo and Anupam Chattopadhyay},
  title = {{Antidistillation is Cheap: Lightweight Distillation-Resistance via Stylistic Corruption}},
  year = {2026},
  eprint = {2611.17772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17772},
}
Figure