MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

Jonathan Nöther, Adish Singla, Goran Radanović

Abstract

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. We formalize this challenge as a Stackelberg security game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary-Meta-Agent (MaMa), a novel algorithm for approximately solving this game and automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.

Introduction

As Large Language Model (LLM) applications transition from single-turn interactions to complex multi-agent systems (MAS), they gain the autonomy to manage finances, use external tools, and modify file systems. However, this increased agency introduces a critical security vulnerability: if even one agent in a chain is compromised—via prompt injection, model failure, or direct exploitation—the entire system's safety can be subverted.

Existing automated design tools for MAS typically prioritize task performance, treating safety as an afterthought often addressed through manual, brittle guardrails. This work introduces MaMa (Meta-Adversary–Meta-Agent), a framework that automates the design of agentic systems to be "robust by construction" through a game-theoretic lens.

Methodology: The Stackelberg Security Game

MaMa formalizes the design process as a Stackelberg security game. In this model, the Meta-Agent (Designer) acts as the leader, proposing a system architecture that includes agent roles, instructions, tool access, and communication protocols. The Meta-Adversary (Attacker) acts as the follower, observing the design and identifying the most damaging way to compromise a subset of agents to minimize the system's safety score.

The MaMa framework: An iterative Stackelberg game between a Meta-Agent designer and a Meta-Adversary.

The MaMa algorithm solves this game through an iterative, LLM-based adversarial search loop:

  1. Propose: The Meta-Agent suggests a system design.
  2. Attack: The Meta-Adversary identifies the strongest attacks against that specific architecture.
  3. Learn: The Meta-Agent receives textual feedback and performance data from these attacks.
  4. Refine: The Meta-Agent updates the architecture to mitigate vulnerabilities while preserving task quality.

Experimental Evaluation

We evaluated MaMa using the BAD-ACTS benchmark across four diverse domains: Travel Planning, Personal Assistant, Financial Article Writing, and Code Generation. These environments test the system's ability to handle sensitive tasks like bank transactions and software documentation under adversarial pressure.

Key Results and Insights

Empirical results demonstrate that MaMa-designed systems significantly outperform baselines in safety while maintaining high task quality.

Performance and safety metrics across four domains over multiple generations of the MaMa algorithm.
  • Architectural Defense: Rather than just tweaking prompts, the Meta-Agent often evolves the system architecture by adding redundant oversight. For example, it introduces "Reviewers," "Auditors," or "Safety Filters" that must approve actions before execution.
  • Safety-Quality Synergy: Interestingly, safety optimizations often improved task quality. The added reviewer agents caught non-adversarial errors, proving that robust design benefits general system reliability.
  • Robust Generalization: Systems designed to withstand an adversary using one LLM (e.g., Qwen) remained robust when the adversary switched to a stronger model (e.g., GPT-4o). Furthermore, systems designed to handle one compromised agent often remained resilient even when multiple agents were subverted.
  • Quantitative Gains: In Travel Planning, MaMa raised the safety score from a baseline of 1.66/5 under attack to approximately 4.0/5. The Attack Success Rate (ASR) dropped to as low as 0.0068, compared to ~0.52 for simpler "Guardian-Agent" baselines.

Limitations and Future Work

While effective, the iterative nature of MaMa is computationally intensive; the experiments conducted cost approximately $72 in API fees over several days. Future research could explore using regression models to predict performance and safety to accelerate the search loop. Additionally, while the current adversary focuses on overwriting agent instructions, future iterations could expand the attack surface to include manipulation of the communication graph or tool definitions.

Summary updated

Abstract

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. We formalize this challenge as a Stackelberg security game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary-Meta-Agent (MaMa), a novel algorithm for approximately solving this game and automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.
BibTeX
@inproceedings{nother2026mama,
  author = {Jonathan Nöther and Adish Singla and Goran Radanović},
  title = {{MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=KR6avDABG8},
}
Figure