Traditional security evaluations often treat threats as static obstacles—like a wall or a fixed filter. In this view, a defender builds a model, and an attacker tries to find a hole in it. However, research across domains ranging from large language models (LLMs) to power grid restoration reveals a fundamental flaw in this perspective: real-world adversaries are not static. They are "rational" agents who observe, learn, and adapt their tactics specifically to bypass the defenses they encounter.
The Static Assumption Fallacy
Most security benchmarks rely on fixed datasets of known attacks. For example, in the realm of LLMs, researchers often test safety by seeing if a model refuses a list of "jailbreak" prompts. As Sahar Abdelnabi and Ahmed Salem demonstrate with the LLMail-Inject dataset, these static evaluations fail to capture the iterative feedback loop of a real attack. A human or AI adversary doesn't just give up after one rejection; they refine their formatting, use social engineering, or switch languages based on the model's response.
This "static assumption" leads to a dangerous overestimation of robustness. Hangsheng Zhang and Jiqiang Liu show in their work on the CARE platform that even sophisticated ensemble defenses—which combine multiple detectors—can collapse when an attacker is allowed to iteratively probe the system for blind spots.
Security as a Stackelberg Game
To address this, researchers are reframing security as a Stackelberg game. In this game-theoretic model, the defender (the "leader") commits to a strategy first, and the attacker (the "follower") observes this strategy before choosing their optimal counter-move.
Sanjay Chawla and Wei Liu 0007 pioneered this approach in Mining adversarial patterns via regularized loss minimization. Instead of training a model on historical data, they use "anticipatory learning." The model essentially asks: "If I deploy this classifier, what is the best way for an adversary to manipulate their data to beat it?" By solving for the Nash equilibrium of this interaction, the defender creates a model that is robust not just to current attacks, but to the optimal future moves of the adversary.
This framework extends beyond software. Anupam Datta and Arunesh Sinha applied similar logic to Audit Games, modeling how insiders adapt their behavior to bypass predictable inspection schedules. By introducing a degree of strategic randomness, defenders can make it mathematically impossible for an attacker to find a "safe" time to commit a violation.
Modeling Multi-Turn and Agentic Interactions
As AI moves from static classification to multi-turn conversation and autonomous agents, the need for dynamic modeling becomes even more acute. Chengxiao Wang and Gagandeep Singh introduced $C^3LLM$ in Quantifying Risks in Multi-turn Conversation with Large Language Models, which models conversations as a Markov process on a semantic graph. This allows them to "certify" safety by calculating the probability that an adaptive adversary can navigate the conversation toward a catastrophic response over several turns.
In multi-agent environments, the challenge is even more complex. Adish Singla and Jonathan Nöther found in their BAD-ACTS benchmark that highly capable models are often more vulnerable to internal manipulation because they can be convinced by an adversarial agent to perform harmful actions through complex, multi-step reasoning.
From Detection to Deception and Continuous Adaptation
If an adversary is going to adapt, the defender must also become dynamic. This has led to two major research directions:
- Cyber Deception: Instead of just blocking attacks, systems like those proposed by Mohamadreza Ahmadi and Nils Jansen in The Partially Observable Games We Play for Cyber Deception use Partially Observable Stochastic Games (POSG) to actively mislead attackers. By feeding the adversary false information, the defender can waste the attacker's resources and learn their intent without exposing the real system.
- Continual Robustness: To prevent "catastrophic forgetting" as attackers change tactics, Ricky Maulana Fajri and Mykola Pechenizkiy developed RoAL. This framework uses Elastic Weight Consolidation to ensure that as a model learns to defend against a new type of attack, it doesn't become vulnerable to the old ones.
The common thread across these 112 papers is a shift in the definition of "robustness." It is no longer enough for a system to be correct on a test set. To be truly secure, a system must be evaluated as a participant in a dynamic, adversarial game, where the only constant is the opponent's willingness to change.