Introduction
In recent years, academic research into AI security has exploded, focusing heavily on sophisticated "adversarial" attacks such as evasion (e.g., fooling image classifiers with subtle perturbations) and data poisoning. However, a significant disconnect exists between these laboratory-based threats and the reality of AI systems deployed in the wild. While researchers explore mathematical optimizations, real-world incidents often stem from traditional cybersecurity failures. This paper analyzes a curated database of 32 real-world AI security incidents to bridge the gap between theory and practice.
Methodology and Systematization
To analyze the landscape of AI security, the authors combined existing incident reports with new data from a survey of 271 AI practitioners. These reports were filtered into a core dataset of 32 distinct malicious incidents. Each incident was categorized using a systematization framework based on three dimensions:
- Target: The specific component under attack—Data (D), the Model (M), or the underlying Infrastructure (Inf).
- Time: Whether the attack occurred during Training (Tr) or Test/Inference (T).
- Goal: The aspect of the "CIA triad" compromised—Confidentiality (C), Integrity (I), or Availability (A).
Incident Analysis and Industry Trends
The data reveals that AI systems are not targeted in a vacuum; they are part of a broader corporate infrastructure. Interestingly, the primary predictor of an attack is not the maturity or age of a specific AI model, but rather the size of the company. Larger organizations, or "big fish," are targeted more frequently regardless of how long their systems have been in production.
Key Findings
1. Infrastructure as the Primary Vulnerability
A major takeaway is that many "AI incidents" do not actually target the neural network's weights or logic. Instead, attackers often exploit the infrastructure supporting the AI. Common examples include crypto-miners abusing free-tier APIs or unauthorized access to cloud storage containing sensitive training data. As shown in the summary below, cybersecurity (cysec) and privacy-related incidents dominate the dataset.
2. The "Boring" Solution to AI Security
Perhaps the most striking result is the avoidability of these incidents. Nearly half (46.9%) of the analyzed cases were deemed preventable through standard cybersecurity and privacy best practices, such as robust access control and proper cloud configuration. Only 18.8% of incidents required advanced, AI-specific scientific research to mitigate.
3. Evasion in the Real World
While academic evasion attacks are often mathematically complex, real-world evasion is more pragmatic. For instance, users might spam a job search platform to reverse-engineer its ranking algorithm. These "low-tech" but effective methods are frequently overlooked in favor of more exotic adversarial research.
Conclusion
The current state of AI security suggests that practitioners should prioritize "boring" security hygiene—securing APIs, managing data access, and hardening infrastructure—before investing in complex adversarial defenses. However, the study also highlights a critical need for better data sharing. Due to NDAs and reputational fears, many incidents remain undocumented. Establishing anonymous, standardized reporting environments is essential for the community to learn from real-world failures and build more resilient AI systems.
Summary updated