When Your AI Becomes a Target: AI Security Incidents and Best Practices

Kathrin Grosse, Lukas Bieringer, Tarek R. Besold, Battista Biggio, Alexandre Alahi

Introduction

In recent years, academic research into AI security has exploded, focusing heavily on sophisticated "adversarial" attacks such as evasion (e.g., fooling image classifiers with subtle perturbations) and data poisoning. However, a significant disconnect exists between these laboratory-based threats and the reality of AI systems deployed in the wild. While researchers explore mathematical optimizations, real-world incidents often stem from traditional cybersecurity failures. This paper analyzes a curated database of 32 real-world AI security incidents to bridge the gap between theory and practice.

Methodology and Systematization

To analyze the landscape of AI security, the authors combined existing incident reports with new data from a survey of 271 AI practitioners. These reports were filtered into a core dataset of 32 distinct malicious incidents. Each incident was categorized using a systematization framework based on three dimensions:

  • Target: The specific component under attack—Data (D), the Model (M), or the underlying Infrastructure (Inf).
  • Time: Whether the attack occurred during Training (Tr) or Test/Inference (T).
  • Goal: The aspect of the "CIA triad" compromised—Confidentiality (C), Integrity (I), or Availability (A).

The data reveals that AI systems are not targeted in a vacuum; they are part of a broader corporate infrastructure. Interestingly, the primary predictor of an attack is not the maturity or age of a specific AI model, but rather the size of the company. Larger organizations, or "big fish," are targeted more frequently regardless of how long their systems have been in production.

Distribution of incidents across categories for 2022 and 2023, showing a higher frequency in larger-scale environments.

Key Findings

1. Infrastructure as the Primary Vulnerability

A major takeaway is that many "AI incidents" do not actually target the neural network's weights or logic. Instead, attackers often exploit the infrastructure supporting the AI. Common examples include crypto-miners abusing free-tier APIs or unauthorized access to cloud storage containing sensitive training data. As shown in the summary below, cybersecurity (cysec) and privacy-related incidents dominate the dataset.

Summary of incident counts categorized by type, target, and security goal, highlighting the prevalence of infrastructure and confidentiality targets.

2. The "Boring" Solution to AI Security

Perhaps the most striking result is the avoidability of these incidents. Nearly half (46.9%) of the analyzed cases were deemed preventable through standard cybersecurity and privacy best practices, such as robust access control and proper cloud configuration. Only 18.8% of incidents required advanced, AI-specific scientific research to mitigate.

3. Evasion in the Real World

While academic evasion attacks are often mathematically complex, real-world evasion is more pragmatic. For instance, users might spam a job search platform to reverse-engineer its ranking algorithm. These "low-tech" but effective methods are frequently overlooked in favor of more exotic adversarial research.

Conclusion

The current state of AI security suggests that practitioners should prioritize "boring" security hygiene—securing APIs, managing data access, and hardening infrastructure—before investing in complex adversarial defenses. However, the study also highlights a critical need for better data sharing. Due to NDAs and reputational fears, many incidents remain undocumented. Establishing anonymous, standardized reporting environments is essential for the community to learn from real-world failures and build more resilient AI systems.

Summary updated

Abstract

In contrast to vast academic efforts to study AI security, few real-world reports of AI security incidents exist. Released incidents prevent a thorough investigation of the attackers' motives, as crucial information about the company and AI application is missing. As a consequence, it often remains unknown how to avoid incidents. We tackle this gap and combine previous reports with freshly collected incidents to a small database of 32 AI security incidents. We analyze the attackers' target and goal, influencing factors, causes, and mitigations. Many incidents stem from non-compliance with best practices in security and privacy-enhancing technologies. In the case of direct AI attacks, access control may provide some mitigation, but there is little scientific work on best practices. Our paper is thus a call for action to address these gaps.
BibTeX
@inproceedings{grosse2024when,
  author = {Kathrin Grosse and Lukas Bieringer and Tarek R. Besold and Battista Biggio and Alexandre Alahi},
  title = {{When Your AI Becomes a Target: AI Security Incidents and Best Practices}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30347},
}

Loading related content...

Figure