A Model Stealing Attack Against Multi-Exit Networks

Li Pan, Lv Peizhuo, Kai Chen, Cai Yuling, Xiang Fan, Zhang Shengzhi

Abstract

Compared to traditional neural networks with a single output channel, a multi-exit network has multiple exits that allow for early outputs from the model's intermediate layers, thus significantly improving computational efficiency while maintaining similar main task accuracy. Existing model stealing attacks can only steal the model's utility while failing to capture its output strategy, i.e., a set of thresholds used to determine from which exit to output. This leads to a significant decrease in computational efficiency for the extracted model, thereby losing the advantage of multi-exit networks. In this paper, we propose the first model stealing attack against multi-exit networks to extract both the model utility and the output strategy. We employ Kernel Density Estimation to analyze the target model's output strategy and use performance loss and strategy loss to guide the training of the extracted model. Furthermore, we design a novel output strategy search algorithm to maximize the consistency between the victim model and the extracted model's output behaviors. In experiments across multiple multi-exit networks and benchmark datasets, our method always achieves accuracy and efficiency closest to the victim models.

Introduction

Traditional deep learning models function like a fixed-length assembly line: every input, regardless of its complexity, must pass through every layer of the network. Multi-Exit Networks (MENs) revolutionize this approach by introducing "off-ramps" or intermediate exits. If the model achieves high confidence at an early stage, it can output a result immediately, significantly reducing computational overhead and energy consumption.

Architecture of a Multi-Exit Network showing intermediate classifiers and output strategies.

However, this efficiency introduces a new security vulnerability. While standard model stealing attacks focus on replicating the final classification logic (utility), they fail to capture the "output strategy"—the specific thresholds that determine when to exit early. Without this strategy, a stolen model becomes a "slow clone" that, while accurate, lacks the speed advantages of the original victim model. This paper introduces the first attack designed to steal both the accuracy and the efficiency of MENs.

Methodology

The proposed attack utilizes a three-stage pipeline to replicate the victim model's behavior:

  1. Estimation Stage (Timing Side-Channel): Because black-box APIs do not disclose which exit was used for a specific query, the researchers use execution time as a side-channel. By applying Kernel Density Estimation (KDE) to the distribution of response times, they can identify temporal boundaries between different exits and label query samples accordingly.
  2. Training Stage (Dual-Loss Optimization): The substitute model is trained using two distinct loss functions: a Performance Loss ($L_P$) to match classification outputs and a Strategy Loss ($L_S$) to force the clone to mimic the victim's exit decisions.
  3. Output Strategy Search: Following training, a specialized search algorithm identifies the optimal confidence thresholds to maximize the consistency (Closeness) between the clone and the victim.

Experimental Results

The attack was evaluated using standard benchmarks including CIFAR10, GTSRB, SVHN, and FashionMNIST. The researchers compared their method against three popular baselines: KnockOff, Noise, and ESA.

Performance comparison of the proposed attack against baseline methods across various models and datasets.

As shown in the experimental data, the proposed method significantly outperforms baselines in maintaining efficiency. While baseline attacks often produced clones that were 40% to 80% slower than the victim, this method's clones remained within 4-15% of the victim's computational cost. Furthermore, the "Closeness" (CLO) metric—measuring how often the clone and victim make the exact same exit decision—was significantly higher for the proposed method, often exceeding 60%. Notably, the KDE-based estimation stage predicted the correct exit number with nearly 100% accuracy across all tested architectures.

Conclusion

This study demonstrates that the efficiency of Multi-Exit Networks is a stealable asset. By leveraging timing side-channels and dual-loss training, attackers can create high-fidelity clones that match both the utility and the strategy of the victim. These findings suggest that future MEN deployments must consider defenses such as adversarial timing jitter or output obfuscation to protect their intellectual property and operational advantages.

Summary updated

Abstract

Compared to traditional neural networks with a single output channel, a multi-exit network has multiple exits that allow for early outputs from the model's intermediate layers, thus significantly improving computational efficiency while maintaining similar main task accuracy. Existing model stealing attacks can only steal the model's utility while failing to capture its output strategy, i.e., a set of thresholds used to determine from which exit to output. This leads to a significant decrease in computational efficiency for the extracted model, thereby losing the advantage of multi-exit networks. In this paper, we propose the first model stealing attack against multi-exit networks to extract both the model utility and the output strategy. We employ Kernel Density Estimation to analyze the target model's output strategy and use performance loss and strategy loss to guide the training of the extracted model. Furthermore, we design a novel output strategy search algorithm to maximize the consistency between the victim model and the extracted model's output behaviors. In experiments across multiple multi-exit networks and benchmark datasets, our method always achieves accuracy and efficiency closest to the victim models.
BibTeX
@misc{pan2023model,
  author = {Li Pan and Lv Peizhuo and Kai Chen and Cai Yuling and Xiang Fan and Zhang Shengzhi},
  title = {{A Model Stealing Attack Against Multi-Exit Networks}},
  year = {2023},
  eprint = {2305.13584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13584},
}
Figure