Security Risks in Deep Learning Implementations

Qixue Xiao, Kang Li, Deyue Zhang, Weilin Xu

Abstract

Advances in deep learning algorithms overshadow their security risk in software implementations. This paper discloses a set of vulnerabilities in popular deep learning frameworks including Caffe, TensorFlow, and Torch. Contrary to the small code size of deep learning models, these deep learning frameworks are complex, and they heavily depend on numerous open source packages. This paper considers the risks caused by these vulnerabilities by studying their impact on common deep learning applications such as voice recognition and image classification. By exploiting these framework implementations, attackers can launch denial-of-service attacks that crash or hang a deep learning application, or control-flow hijacking attacks that lead to either system compromise or recognition evasions. The goal of this paper is to draw attention to software implementations and call for community collaborative effort to improve security of deep learning frameworks.

Introduction

While the AI community is fixated on "adversarial examples" that trick a model’s logic, this research highlights a more fundamental danger: the massive, complex software stacks running these models are riddled with traditional security vulnerabilities. By exploiting bugs in frameworks like TensorFlow and Caffe, or their dependencies like OpenCV, attackers can crash applications, force misclassifications, or even gain full remote control of a system using nothing more than a specially crafted image file.

Most research into Deep Learning (DL) security focuses on the "math"—how to perturb pixels so a stop sign looks like a speed limit sign. However, we argue that the industry is largely ignoring the "plumbing." Deep learning frameworks are not just simple math engines; they are behemoths of code. TensorFlow, for instance, contains nearly a million lines of code and relies on nearly a hundred third-party packages for tasks like image processing and matrix multiplication.

The DL Software Stack and Attack Surface

This complexity creates a massive "attack surface." If there is a classic software bug (like a buffer overflow) in the code that loads an image before the neural network even sees it, an attacker can bypass the model's logic entirely to compromise the underlying system.

The multi-layered software stack of deep learning applications and their extensive dependencies.

As shown in Figure 1, the DL stack consists of three main layers: the application layer (logic, model, data), the framework layer (e.g., TensorFlow, Caffe), and the dependency layer (e.g., NumPy, OpenCV). A vulnerability in any of these lower-level dependencies effectively becomes a vulnerability in the entire application. TensorFlow has over 887K lines of code, while Caffe relies on 137 external libraries, illustrating the scale of the risk.

Methodology

The researchers performed a security audit of three dominant DL frameworks: Caffe, TensorFlow, and Torch. Their approach involved:

  • Complexity Analysis: Using tools like cloc and ldd to quantify the sheer volume of code and external dependencies.
  • Vulnerability Discovery: Applying traditional bug-finding techniques, such as fuzzing, to identify implementation flaws.
  • Attack Surface Mapping: Categorizing how an attacker might interact with a DL system through malformed operational inputs, training data (poisoning), and shared models.
  • Proof-of-Concept (PoC) Exploitation: Crafting specific inputs to trigger discovered bugs and demonstrate real-world consequences like Denial of Service (DoS) and remote shell access.

Key Results and Case Studies

The audit resulted in the discovery of over 15 new vulnerabilities (CVEs) confirmed by developers. The threats demonstrated fall into three primary tiers:

  1. Denial of Service (DoS): A bug in numpy.pad (used by TensorFlow) could be triggered by an empty vector, causing an infinite loop that hangs the application.
  2. Evasion: Using a malformed BMP, the researchers triggered a memory corruption bug in OpenCV that allowed them to force a "Bulldog" to be classified as a "Flying Pig."
  3. System Compromise: They successfully demonstrated a remote shell exploit. By feeding a crafted image to a Caffe-based service, they gained command-line access to the host server.
A representative image used in the "Bulldog" proof-of-concept attacks to trigger implementation vulnerabilities.

The "Bulldog" PoC (Figure 2) illustrates how malformed inputs can bypass algorithmic logic. By triggering an out-of-bounds write bug in the image-loading code, an attacker can manually overwrite the memory location where the classification result is stored, achieving evasion without needing to optimize gradients.

Conclusion and Future Work

Current fuzzing tools are not optimized for the unique execution flow of deep learning models. Because neural networks usually execute the same layers regardless of input values, the "execution path" rarely changes, making it hard for traditional fuzzers to find edge cases in the model logic itself. Future work must focus on logic-aware detection methods and distinguishing between "insufficient training" and "intended manipulation." This paper serves as a call for community collaborative effort to improve the security of deep learning frameworks at the implementation level.

Summary updated

Abstract

Advances in deep learning algorithms overshadow their security risk in software implementations. This paper discloses a set of vulnerabilities in popular deep learning frameworks including Caffe, TensorFlow, and Torch. Contrary to the small code size of deep learning models, these deep learning frameworks are complex, and they heavily depend on numerous open source packages. This paper considers the risks caused by these vulnerabilities by studying their impact on common deep learning applications such as voice recognition and image classification. By exploiting these framework implementations, attackers can launch denial-of-service attacks that crash or hang a deep learning application, or control-flow hijacking attacks that lead to either system compromise or recognition evasions. The goal of this paper is to draw attention to software implementations and call for community collaborative effort to improve security of deep learning frameworks.
BibTeX
@misc{xiao2017security,
  author = {Qixue Xiao and Kang Li and Deyue Zhang and Weilin Xu},
  title = {{Security Risks in Deep Learning Implementations}},
  year = {2017},
  eprint = {1711.11008},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.11008},
}
Figure