SoK: A Systems Perspective on Compound AI Threats and Countermeasures

Sarbartha Banerjee, Prateek Sahu, Mulong Luo, Anjo Vahldiek-Oberwagner, Neeraja J. Yadwadkar, Mohit TiwariarXiv 2024 · OpenReview

TL;DR

This paper provides a comprehensive "State of the Knowledge" (SoK) on the security threats and defenses in complex, multi-component AI systems (compound AI). It argues that understanding and mitigating these threats requires a holistic, cross-layer perspective, moving beyond isolated attacks on algorithms, software, or hardware.

What Problem Does This Paper Address?

Modern AI systems, especially those leveraging Large Language Models (LLMs), are increasingly built as "compound AI" pipelines. These pipelines integrate multiple AI models, databases, and various software and hardware components. This complexity significantly expands the attack surface. While prior research has focused on individual vulnerabilities (e.g., prompt injection, hardware bit-flips), these studies often examine them in isolation. This paper addresses the critical gap of understanding how attackers can combine vulnerabilities across different layers (application, software, hardware) to launch more powerful, end-to-end attacks with fewer assumptions about their capabilities. This holistic understanding is crucial for designing secure and robust AI deployments.

Methods

The paper employs a systematic approach to survey and categorize existing threats and defenses in compound AI systems. Key methodological contributions include:

  • Cross-Layer Attack Analysis: The authors analyze how vulnerabilities in one layer (e.g., software supply chain) can be leveraged to amplify attacks in another (e.g., hardware timing attacks).
  • Mitre ATT&CK Framework Mapping: They map various AI/ML system and hardware attacks to the well-established Mitre ATT&CK framework. This provides a structured way to understand attacker tactics, techniques, and procedures (TTPs) in the context of AI systems.
  • Threat Model Categorization: The paper defines four threat models (remote software access, privileged software access, digital hardware access, physical hardware access) to characterize attacker capabilities and their increasing severity.
  • Asset Identification: They identify critical "secret assets" within compound AI systems, such as model weights, training data, and inference data, which are targets for attackers.
  • Comprehensive Literature Survey: The paper synthesizes a vast body of research on software vulnerabilities, hardware attacks, and existing countermeasures, organizing them by layer and attack type.

Datasets

Not discussed in the paper. The paper focuses on systematizing existing knowledge and identifying attack vectors and defenses rather than introducing new datasets or benchmarks.

Key Insights and Ideas

  • The Power of Combination: Isolated attacks, which might seem minor, become significantly more potent when combined across different layers of the AI stack. For example, a software vulnerability might enable an attacker to gain the necessary access to perform a more impactful hardware attack.
  • Beyond Algorithmic Attacks: While algorithmic attacks (like membership inference) are well-studied, the paper highlights that system-level vulnerabilities (in software and hardware) are crucial enablers for these attacks, often relaxing their stringent assumptions.
  • Mitre ATT&CK is a Useful Lens: Mapping AI attacks to the Mitre ATT&CK framework provides a standardized language and structure for understanding attacker behavior and developing defense strategies.
  • Hardware is Not Immune: Hardware vulnerabilities, from timing side-channels to bit-flips, are increasingly relevant and can be exploited to compromise AI systems, especially in multi-tenant environments.
  • Holistic Defense is Essential: Effective security for compound AI cannot be achieved by focusing on individual components. A comprehensive, cross-layer defense strategy is necessary.

Key Results

The paper doesn't present new empirical results from experiments but rather synthesizes existing findings. Its key "results" are in the form of:

  • A Systematized Taxonomy: A structured categorization of software and hardware attacks applicable to compound AI systems.
  • Cross-Layer Attack Case Studies: Four illustrative examples of how combining algorithmic, software, and hardware techniques can lead to powerful end-to-end attacks (e.g., model extraction, bitflip for degradation, covert channels, knowledge tampering).
  • A Comprehensive Defense Overview: A survey of existing countermeasures for software and hardware vulnerabilities, highlighting their limitations in the context of compound AI.
  • Identification of Open Research Questions: The paper points to areas needing further investigation, such as developing quantitative metrics for AI attack severity and creating robust cross-layer defense mechanisms.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that many defenses are designed for specific hardware platforms (e.g., CPUs) and their applicability to heterogeneous AI hardware (GPUs, TPUs, FPGAs) needs further investigation.
  • The complexity of compound AI systems makes formal verification and exhaustive fuzzing challenging.
  • The paper highlights the difficulty in establishing clear alignment between user privacy policies and system security requirements across multiple distrustful entities.
  • Future Work:
    • Developing quantitative metrics for AI attack severity, considering asset sensitivity, attacker capabilities, and overall impact.
  • Designing robust cross-layer defenses that integrate algorithmic, software, and hardware security measures.
  • Exploring the compositionality of differential privacy (DP) budgets in compound AI systems.
  • Investigating information flow control (IFC) mechanisms that span across software and hardware layers.
  • Developing standardized security frameworks and attestation mechanisms for AI systems.
  • Building a robust compound AI system by establishing clear trust relationships and cross-stack alignment of security requirements.

Summary updated

Abstract

Large language models (LLMs) used across enterprises often use proprietary models and operate on sensitive inputs and data. The wide range of attack vectors identified in prior research - targeting various software and hardware components used in training and inference - makes it extremely challenging to enforce confidentiality and integrity policies. As we advance towards constructing compound AI inference pipelines that integrate multiple large language models (LLMs), the attack surfaces expand significantly. Attackers now focus on the AI algorithms as well as the software and hardware components associated with these systems. While current research often examines these elements in isolation, we find that combining cross-layer attack observations can enable powerful end-to-end attacks with minimal assumptions about the threat model. Given, the sheer number of existing attacks at each layer, we need a holistic and systemized understanding of different attack vectors at each layer. This SoK discusses different software and hardware attacks applicable to compound AI systems and demonstrates how combining multiple attack mechanisms can reduce the threat model assumptions required for an isolated attack. Next, we systematize the ML attacks in lines with the Mitre Att&ck framework to better position each attack based on the threat model. Finally, we outline the existing countermeasures for both software and hardware layers and discuss the necessity of a comprehensive defense strategy to enable the secure and high-performance deployment of compound AI systems.
BibTeX
@misc{banerjee2024sok,
  author = {Sarbartha Banerjee and Prateek Sahu and Mulong Luo and Anjo Vahldiek-Oberwagner and Neeraja J. Yadwadkar and Mohit Tiwari},
  title = {{SoK: A Systems Perspective on Compound AI Threats and Countermeasures}},
  year = {2024},
  eprint = {2411.13459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13459},
}
Figure