TL;DR
This paper provides a comprehensive "State of the Knowledge" (SoK) on the security threats and defenses in complex, multi-component AI systems (compound AI). It argues that understanding and mitigating these threats requires a holistic, cross-layer perspective, moving beyond isolated attacks on algorithms, software, or hardware.
What Problem Does This Paper Address?
Modern AI systems, especially those leveraging Large Language Models (LLMs), are increasingly built as "compound AI" pipelines. These pipelines integrate multiple AI models, databases, and various software and hardware components. This complexity significantly expands the attack surface. While prior research has focused on individual vulnerabilities (e.g., prompt injection, hardware bit-flips), these studies often examine them in isolation. This paper addresses the critical gap of understanding how attackers can combine vulnerabilities across different layers (application, software, hardware) to launch more powerful, end-to-end attacks with fewer assumptions about their capabilities. This holistic understanding is crucial for designing secure and robust AI deployments.
Methods
The paper employs a systematic approach to survey and categorize existing threats and defenses in compound AI systems. Key methodological contributions include:
- Cross-Layer Attack Analysis: The authors analyze how vulnerabilities in one layer (e.g., software supply chain) can be leveraged to amplify attacks in another (e.g., hardware timing attacks).
- Mitre ATT&CK Framework Mapping: They map various AI/ML system and hardware attacks to the well-established Mitre ATT&CK framework. This provides a structured way to understand attacker tactics, techniques, and procedures (TTPs) in the context of AI systems.
- Threat Model Categorization: The paper defines four threat models (remote software access, privileged software access, digital hardware access, physical hardware access) to characterize attacker capabilities and their increasing severity.
- Asset Identification: They identify critical "secret assets" within compound AI systems, such as model weights, training data, and inference data, which are targets for attackers.
- Comprehensive Literature Survey: The paper synthesizes a vast body of research on software vulnerabilities, hardware attacks, and existing countermeasures, organizing them by layer and attack type.
Datasets
Not discussed in the paper. The paper focuses on systematizing existing knowledge and identifying attack vectors and defenses rather than introducing new datasets or benchmarks.
Key Insights and Ideas
- The Power of Combination: Isolated attacks, which might seem minor, become significantly more potent when combined across different layers of the AI stack. For example, a software vulnerability might enable an attacker to gain the necessary access to perform a more impactful hardware attack.
- Beyond Algorithmic Attacks: While algorithmic attacks (like membership inference) are well-studied, the paper highlights that system-level vulnerabilities (in software and hardware) are crucial enablers for these attacks, often relaxing their stringent assumptions.
- Mitre ATT&CK is a Useful Lens: Mapping AI attacks to the Mitre ATT&CK framework provides a standardized language and structure for understanding attacker behavior and developing defense strategies.
- Hardware is Not Immune: Hardware vulnerabilities, from timing side-channels to bit-flips, are increasingly relevant and can be exploited to compromise AI systems, especially in multi-tenant environments.
- Holistic Defense is Essential: Effective security for compound AI cannot be achieved by focusing on individual components. A comprehensive, cross-layer defense strategy is necessary.
Key Results
The paper doesn't present new empirical results from experiments but rather synthesizes existing findings. Its key "results" are in the form of:
- A Systematized Taxonomy: A structured categorization of software and hardware attacks applicable to compound AI systems.
- Cross-Layer Attack Case Studies: Four illustrative examples of how combining algorithmic, software, and hardware techniques can lead to powerful end-to-end attacks (e.g., model extraction, bitflip for degradation, covert channels, knowledge tampering).
- A Comprehensive Defense Overview: A survey of existing countermeasures for software and hardware vulnerabilities, highlighting their limitations in the context of compound AI.
- Identification of Open Research Questions: The paper points to areas needing further investigation, such as developing quantitative metrics for AI attack severity and creating robust cross-layer defense mechanisms.
Limitations and Future Work
- Limitations:
- The paper acknowledges that many defenses are designed for specific hardware platforms (e.g., CPUs) and their applicability to heterogeneous AI hardware (GPUs, TPUs, FPGAs) needs further investigation.
- The complexity of compound AI systems makes formal verification and exhaustive fuzzing challenging.
- The paper highlights the difficulty in establishing clear alignment between user privacy policies and system security requirements across multiple distrustful entities.
- Future Work:
- Developing quantitative metrics for AI attack severity, considering asset sensitivity, attacker capabilities, and overall impact.
- Designing robust cross-layer defenses that integrate algorithmic, software, and hardware security measures.
- Exploring the compositionality of differential privacy (DP) budgets in compound AI systems.
- Investigating information flow control (IFC) mechanisms that span across software and hardware layers.
- Developing standardized security frameworks and attestation mechanisms for AI systems.
- Building a robust compound AI system by establishing clear trust relationships and cross-stack alignment of security requirements.
Summary updated