This position paper from researchers at the University of Melbourne and DST Group is a wake-up call for the Adversarial Machine Learning (AML) community. It argues that the field has become trapped in a "metrological ball-and-chain": the obsession with the $\epsilon$-ball (epsilon-ball).
TL;DR
The "small epsilon-ball" paradigm—focusing on tiny, imperceptible perturbations to fool AI—is a misleading proxy for real-world risk. The authors argue that this fixation ignores the most dangerous threats of the Generative AI era, such as high-fidelity synthetic deepfakes and supply-chain attacks, and call for a shift toward a "Systemic Risk Framework" grounded in economics and actual harm.
What Problem Does This Paper Address?
Since the discovery that neural networks could be subverted by nearly invisible pixel changes, AML research has centered on the $\ell_p$ norm. This framework assumes that an attacker’s goal is to find the smallest possible change ($\epsilon$) that flips a model's prediction while remaining undetected by humans.
The authors argue this is a "metrological trap." In the lab, these attacks are mathematically elegant. In the real world, they often collapse. If you have to shine a visible laser at a self-driving car's camera to execute an "imperceptible" attack, the attack isn't actually stealthy—it's glaringly obvious. This disconnect creates a false sense of security: we are building robust defenses against "mathematical curiosities" while leaving systems vulnerable to the path of least resistance, such as social engineering or generative synthesis.
Methods
This is a position paper supported by a large-scale meta-analysis of the current research landscape. The authors:
- Analyzed 4311 papers from top-tier conferences (ICML, ICLR, and NeurIPS) spanning 2022–2025.
- Used LLM-assisted categorization (via Google Gemini) to identify the primary threat models and constraints used in these papers.
- Proposed a "Systemic Risk Framework" consisting of four pillars: Ecological Realizability, Cross-Perceptual Visibility, Distributional Authenticity, and the Harm Function.
Datasets
The "dataset" for this study was the corpus of 4311 peer-reviewed AML papers. The authors used the OpenReview API to collect this data and validated their LLM-based classification with a blind human review of a random sample.
Key Insights and Ideas
- The Empty Room Paradox: Many "stealthy" attacks assume an environment devoid of observers. For example, an adversarial audio command hidden in white noise is mathematically "small," but a human would immediately notice a speaker suddenly broadcasting static. If the room is empty, the attacker doesn't need stealth; if it's occupied, the "stealthy" mechanism itself gives them away.
- The Generative Shift: Classical AML assumes attackers perturb a benign input ($x + \delta$). Generative AI allows attackers to sample directly from the data manifold ($g \in \mathcal{M}$). A deepfake doesn't need to be "near" a specific original image to be dangerous; it just needs to be plausible.
- Adversarial Economics: Attackers are rational actors. If a \$5 phishing email achieves the same goal as a \$50,000 white-box gradient-based attack, they will choose the email every time. Current research ignores the "Cost-per-success" metric.
- Metric Relativity: Small changes in pixel space can be semantically huge, while massive pixel changes (like rotating an image 1 degree or turning it grayscale) are semantically irrelevant to humans. The $\ell_p$ norm fails to capture this reality.
Key Results
The meta-analysis confirms that the $\epsilon$-ball remains the dominant orthodoxy in the field.
As seen in Table 1, 31% of all analyzed papers explicitly use the $\epsilon$ threat model as their primary focus. Even more telling, the authors note that the majority of papers in the "Poisoning," "Jailbreaking," and "Defense" categories still rely on bounding perturbations with small epsilon values. This suggests that even when researchers move to newer domains like LLMs, they often drag the old $\ell_p$ baggage with them.
Limitations and Future Work
The authors acknowledge that constructing benchmarks that fully capture "real-world context" is an impossible task. However, they argue that "completely eschewing these factors should not be the answer either."
They propose a three-pillar agenda for the future:
- Broadened Benchmarking: Moving beyond model architectures to consider the entire pipeline (sensors, noise, compression, and downstream context).
- Cost Contextualization: Quantifying the socio-economic cost ($C_{attack}$) required for an intervention to succeed.
- Measuring Operational Impacts: Treating downstream harm—financial loss, physical injury, or loss of trust—as a first-class evaluation criterion.
The paper concludes with a call for the community to abandon the $\epsilon$-ball as its default proxy for risk and instead align research with the messy, expensive, and high-stakes reality of deployed AI systems.
Summary updated