Adversarial Risk in the Generative AI Era Necessitates Dropping the Small Epsilon Ball

Andrew C. Cullen, Neil G. Marchant, Paul Montague, Jiani Xie, Benjamin I. P. RubinsteinarXiv 2026 · arXiv

This position paper from researchers at the University of Melbourne and DST Group is a wake-up call for the Adversarial Machine Learning (AML) community. It argues that the field has become trapped in a "metrological ball-and-chain": the obsession with the $\epsilon$-ball (epsilon-ball).

TL;DR

The "small epsilon-ball" paradigm—focusing on tiny, imperceptible perturbations to fool AI—is a misleading proxy for real-world risk. The authors argue that this fixation ignores the most dangerous threats of the Generative AI era, such as high-fidelity synthetic deepfakes and supply-chain attacks, and call for a shift toward a "Systemic Risk Framework" grounded in economics and actual harm.

What Problem Does This Paper Address?

Since the discovery that neural networks could be subverted by nearly invisible pixel changes, AML research has centered on the $\ell_p$ norm. This framework assumes that an attacker’s goal is to find the smallest possible change ($\epsilon$) that flips a model's prediction while remaining undetected by humans.

The authors argue this is a "metrological trap." In the lab, these attacks are mathematically elegant. In the real world, they often collapse. If you have to shine a visible laser at a self-driving car's camera to execute an "imperceptible" attack, the attack isn't actually stealthy—it's glaringly obvious. This disconnect creates a false sense of security: we are building robust defenses against "mathematical curiosities" while leaving systems vulnerable to the path of least resistance, such as social engineering or generative synthesis.

Methods

This is a position paper supported by a large-scale meta-analysis of the current research landscape. The authors:

  1. Analyzed 4311 papers from top-tier conferences (ICML, ICLR, and NeurIPS) spanning 2022–2025.
  2. Used LLM-assisted categorization (via Google Gemini) to identify the primary threat models and constraints used in these papers.
  3. Proposed a "Systemic Risk Framework" consisting of four pillars: Ecological Realizability, Cross-Perceptual Visibility, Distributional Authenticity, and the Harm Function.

Datasets

The "dataset" for this study was the corpus of 4311 peer-reviewed AML papers. The authors used the OpenReview API to collect this data and validated their LLM-based classification with a blind human review of a random sample.

Key Insights and Ideas

  • The Empty Room Paradox: Many "stealthy" attacks assume an environment devoid of observers. For example, an adversarial audio command hidden in white noise is mathematically "small," but a human would immediately notice a speaker suddenly broadcasting static. If the room is empty, the attacker doesn't need stealth; if it's occupied, the "stealthy" mechanism itself gives them away.
  • The Generative Shift: Classical AML assumes attackers perturb a benign input ($x + \delta$). Generative AI allows attackers to sample directly from the data manifold ($g \in \mathcal{M}$). A deepfake doesn't need to be "near" a specific original image to be dangerous; it just needs to be plausible.
  • Adversarial Economics: Attackers are rational actors. If a \$5 phishing email achieves the same goal as a \$50,000 white-box gradient-based attack, they will choose the email every time. Current research ignores the "Cost-per-success" metric.
  • Metric Relativity: Small changes in pixel space can be semantically huge, while massive pixel changes (like rotating an image 1 degree or turning it grayscale) are semantically irrelevant to humans. The $\ell_p$ norm fails to capture this reality.

Key Results

The meta-analysis confirms that the $\epsilon$-ball remains the dominant orthodoxy in the field.

Table 1: Threat frequency from 4311 Adversarial ML papers from ICML/ICLR/NeurIPS over 2022– 2025, categorized by primary threat model (all entries are distinct) following Appendix D. Ab- breviations: ϵ : small metric perturbations, Pois.: Poisoning, Jail.: Jailbreak, Sem.: Semantic, Def.: Defense. Note: the majority of Poisoning, Jail- breaking, Patch, and Defense papers employ an ϵ threat model (see Section 2 and Appendix D).

As seen in Table 1, 31% of all analyzed papers explicitly use the $\epsilon$ threat model as their primary focus. Even more telling, the authors note that the majority of papers in the "Poisoning," "Jailbreaking," and "Defense" categories still rely on bounding perturbations with small epsilon values. This suggests that even when researchers move to newer domains like LLMs, they often drag the old $\ell_p$ baggage with them.

Limitations and Future Work

The authors acknowledge that constructing benchmarks that fully capture "real-world context" is an impossible task. However, they argue that "completely eschewing these factors should not be the answer either."

They propose a three-pillar agenda for the future:

  1. Broadened Benchmarking: Moving beyond model architectures to consider the entire pipeline (sensors, noise, compression, and downstream context).
  2. Cost Contextualization: Quantifying the socio-economic cost ($C_{attack}$) required for an intervention to succeed.
  3. Measuring Operational Impacts: Treating downstream harm—financial loss, physical injury, or loss of trust—as a first-class evaluation criterion.

The paper concludes with a call for the community to abandon the $\epsilon$-ball as its default proxy for risk and instead align research with the messy, expensive, and high-stakes reality of deployed AI systems.

Summary updated

Abstract

Adversarial Machine Learning research is currently centered on the search for the epsilon-ball: minimal perturbations designed to fool models while remaining imperceptible to humans. We argue that this fixation on $\ell_p$ norms is a metrological trap that communicates a distorted perception of risk to defenders, one that is unaligned with real-world system vulnerabilities. This paradigm misleads defenders into optimizing for narrow, mathematical robustness at the expense of systemic security, while simultaneously incentivizing attackers to exploit non-overlapping alternative adversarial pathways. Ultimately, we argue that the dominance of the $\ell_p$ ball does not just fail to reduce harm---it may actively facilitate it by communicating a false sense of security against the wrong threats.
BibTeX
@misc{cullen2026adversarial,
  author = {Andrew C. Cullen and Neil G. Marchant and Paul Montague and Jiani Xie and Benjamin I. P. Rubinstein},
  title = {{Adversarial Risk in the Generative AI Era Necessitates Dropping the Small Epsilon Ball}},
  year = {2026},
  eprint = {2611.14594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14594},
}
Figure