Dual-Use Risks in Adversarial Machine Learning

35 concepts32 papers2 proposals

At its core, adversarial machine learning is an optimization problem. To find an adversarial example, an algorithm navigates a model's high-dimensional loss landscape to locate a minute perturbation that fundamentally alters the model's output. Because mathematics is indifferent to intent, this optimization process is inherently symmetrical. A tool designed to map out vulnerabilities so they can be patched can just as easily be used to exploit them.

This symmetry creates a profound dual-use dilemma. In AI safety and security research, the line between a defensive shield and an offensive weapon is often nothing more than a change in perspective.


Defensive Tools as Offensive Blueprints

The most direct manifestation of the dual-use dilemma occurs when frameworks designed to optimize defender strategies are inverted by adversaries. To defend a system strategically, a defender must simulate potential attacks and identify weak points. However, if an adversary gains access to these simulation tools, the defender's blueprint becomes the attacker's roadmap.

Consider game-theoretic simulations used to secure physical or digital infrastructure. As shown by Noah Krever et al. in their work on the GUARD framework, high-fidelity environments that model real-world geospatial and demographic data are essential for stress-testing defensive algorithms. Yet, the authors explicitly note the accompanying dual-use risk: adversaries can exploit these exact simulation tools to model defender behavior, predict responses, and systematically discover tactics to bypass security measures.

This inversion applies to language models as well. In ARMOR, Zhengyue Zhao et al. design a "System-2" defensive framework that protects Large Language Models (LLMs) by explicitly reverse-engineering an attacker's strategy and extracting hidden malicious intent. While this meticulous reasoning drastically reduces attack success rates, the underlying mechanism is a double-edged sword. The capability to decompose and deeply understand complex attacks can be repurposed to engineer more sophisticated, un-filterable adversarial prompts.

Similarly, automated alignment techniques can inadvertently train highly capable saboteurs. Mickel Liu et al. introduce Self-RedTeam, an online self-play reinforcement learning framework that co-evolves an LLM to be both a more creative attacker and a more robust defender. While this zero-sum game successfully hardens the model against manipulation, the authors emphasize that developing advanced red-teaming capabilities within the model introduces severe dual-use risks. If an end-user in a production environment manages to trigger the model's highly optimized adversarial persona, the defense mechanism itself becomes the threat generator.

Defensive Optimization                     Offensive Inversion
       │                                          │
       ▼                                          ▼
Simulate vulnerabilities  ───────────────►  Map out target weaknesses
Reverse-engineer attacks  ───────────────►  Synthesize un-filterable prompts
Co-evolve red-teamers     ───────────────►  Deploy highly creative attackers

The Double-Edged Sword of Safety Benchmarks

A cornerstone of scientific progress in AI safety is evaluation. To know if a model is safe, researchers must test it against standardized, rigorous attack benchmarks. However, publishing a high-quality benchmark often means publishing a curated catalog of functional exploits.

This tension is highly visible in the evaluation of frontier LLMs. Jialin Song et al. developed MultiBreak, a large-scale benchmark designed to evaluate model resilience against multi-turn jailbreak attacks. While MultiBreak is vital for identifying deep-seated vulnerabilities in conversational models, the dataset itself contains thousands of highly effective, diverse jailbreak prompts. As a result, the authors note that access must be restricted to prevent malicious actors from weaponizing the benchmark to facilitate real-world attacks.

The dual-use risk intensifies as adversarial generation techniques become more sophisticated. For instance, Haotian Xue et al. introduce Diff-PGD, a framework that leverages the natural image prior of diffusion models to generate highly stealthy adversarial samples. While intended for defensive hardening, these images are virtually indistinguishable from natural images to humans and highly transferable across different AI architectures, making them incredibly potent tools for malicious exploitation.

Furthermore, as adversarial frameworks advance from simple digital domains (like images) to complex, structured domains, the potential real-world harm escalates. Zhengjie Zhou et al. demonstrate this in Ripple Perturbations Through Structure, where they generate adversarial attacks on heterogeneous tabular data that maintain strict logical and semantic consistency. If an attack framework can alter financial or healthcare records while keeping the data plausible and structurally coherent, it poses a severe dual-use risk that can be weaponized against critical infrastructure unless paired immediately with structure-aware defenses.

This constant overlap between research and exploitation complicates the legal and institutional landscape of AI safety. As Shayne Longpre et al. argue in Position: A Safe Harbor for AI Evaluation and Red Teaming, safety researchers require broad legal and technical protections to evaluate models without fear of account bans or lawsuits. Yet, the authors acknowledge the inherent friction: these same protections could be exploited by malicious actors to probe systems for vulnerabilities under the guise of good-faith research.


Reframing the Adversary for Social Good

If defensive tools can be inverted for harm, can offensive tools be pacified for social good? Because adversarial optimization is mathematically neutral, the concept of an "attack" can be entirely decoupled from malicious intent.

Historically, the adversarial ML community has operated under a rigid binary: defenders are benevolent, and attackers are malicious. Kendra Albert et al. challenge this paradigm in Adversarial for Good?, arguing that treating model robustness as an inherent good blinds researchers to the beneficial applications of adversarial attacks. For individuals resisting invasive automated surveillance or attempting to rectify systemic algorithmic bias, adversarial perturbations offer a technical mechanism for subversion and self-defense.

                          ┌──────────────────────────┐
                          │ Adversarial Optimization │
                          └────────────┬─────────────┘
                                       │
                ┌──────────────────────┴──────────────────────┐
                ▼                                             ▼
     [ Malicious Exploitation ]                     [ Socially Beneficial ]
  - Bypassing security filters                  - Data cloaking for privacy
  - Evading malware detection                   - Resisting invasive surveillance
  - Manipulating financial data                 - Auditing and rectifying bias

Shawqi Al-Maliki et al. formalize this perspective with their AdvML4G framework, which reframes the adversary as an ally. By repurposing the brittleness of deep neural networks, researchers can implement data-cloaking techniques that protect user privacy against unauthorized scraping. However, the authors note that this optimization remains fundamentally dual-use: a tool designed to cloak personal photos to protect privacy uses the exact same technical vectors that a malicious actor might use to evade necessary security filters or content moderation systems.

When stripped of its adversarial framing, a "benign attack" simply exploits a model's reliance on non-semantic features to achieve a constructive outcome. As Jitao Sang et al. demonstrate in Benign Adversarial Attack: Tricking Models for Goodness, these perturbations can be used to construct an "Adversarial Turing Test" to differentiate between human and machine intelligence, or to force a system to reject unauthorized applications of a model.


Institutionalizing Dual-Use Awareness

Because the technical mechanisms of adversarial machine learning cannot be segregated into "safe" and "unsafe" categories, the research community must rely on educational, ethical, and structural frameworks to manage these risks.

In education, this requires moving ethics from an elective afterthought to a core component of technical training. Alexandra Schofield and Emily M. Bender outline a practical curriculum framework in Integrating Ethics into the NLP Curriculum. They introduce structured brainstorming sessions specifically designed to identify dual-use scenarios, forcing students to analyze how benign classifiers or language models could be repurposed for surveillance, censorship, or automated harm before deployment.

Furthermore, understanding these risks requires looking beyond pure mathematics to the human systems that deploy them. Alice Qian Zhang et al. argue in The Human Factor in AI Red Teaming that AI red teaming is fundamentally a human labor process. By drawing on historical precedents from military scenario testing and traditional cybersecurity, the authors illustrate how adversarial methodologies shift when they are applied to social harms rather than technical bugs.

Ultimately, the dual-use nature of adversarial machine learning means that security cannot be achieved through obscurity or unilateral defense. Because every defensive advancement yields a more sophisticated method of attack, the field must navigate a continuous, dynamic equilibrium—balancing the open disclosure necessary for scientific evaluation against the risk of providing a turnkey toolkit for adversaries.

Key Foundational Work

Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attacks (2021) — contributed 2 concept elements to this direction.
Introduction to the Minitrack on Cyber Operations, Defence, and Forensics (2021) — contributed 1 concept elements to this direction.
Robustifying $\ell_\infty$ Adversarial Training to the Union of Perturbation Models (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_potential_for_misuse_f78bb014,
  title = {potential for misuse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/potential-for-misuse/rdr_f78bb014e1b548a4a2f6bf207af71078},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_potential_for_misuse_f78bb014_v1,
  title = {potential for misuse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/potential-for-misuse/rdr_f78bb014e1b548a4a2f6bf207af71078/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake