Traditional adversarial machine learning often treats security as a single-turn game. In static tasks like image classification, an adversary injects a microscopic perturbation into an input, flips the model's prediction, and wins. This approach is fundamentally myopic: it optimizes for instantaneous error at a single point in time.
However, real-world systems—such as autonomous vehicles, robotics, cyber-physical infrastructure, and security patrols—are sequential. They operate in closed-loop environments where an action taken at time $t$ alters the state of the world at time $t+1$. In these dynamic settings, myopic attacks frequently fail. A single-step perturbation might cause a temporary glitch, but the system's feedback loop or safety filters will often correct the error and recover.
To truly subvert a sequential system, an adversary must shift from instantaneous error maximization to long-horizon strategic planning. Instead of trying to break the system immediately, a strategic adversary plans a coordinated sequence of minor interventions over time, steering the target into an irreversible, catastrophic failure state many steps into the future.
The Myopia Trap in Sequential Systems
Why do myopic attacks fall short in dynamic environments? In a sequential decision-making process, such as a Markov Decision Process (MDP), current actions influence future trajectories. As noted by Lin et al., reinforcement learning (RL) agents operate within tight sequential dependencies, meaning that isolated, step-by-step perturbations rarely cause systemic failure.
Despite this, many security frameworks and attack models still rely on single-step optimizations due to the computational simplicity of real-time execution. This limitation is highlighted by Hallyburton et al. in the context of nonlinear estimation and by Arnström et al. in safety filters. These short-sighted strategies can even backfire; Yu et al. demonstrate that myopic heuristics (such as targeting states where the agent has high confidence) can inadvertently redirect a sub-optimal agent onto a more effective trajectory, accidentally helping the victim.
Myopic Attack:
[State 0] --(Perturb)--> [Immediate Error] --> System Recovers in Next Step
Strategic Attack:
[State 0] --(Subtle Perturb)--> [State 1] --> [State 2] --> ... --> [Catastrophic Failure State]
This myopic assumption creates a dangerous blind spot for defenders. In security games, if a defender assumes an adversary will always act greedily to maximize immediate rewards, they become highly vulnerable to strategic deception. As shown by Nguyen et al., Zheng et al., and Li et al., forward-looking adversaries will intentionally choose sub-optimal targets or delay attacks to mislead the defender's learning process, setting up a much larger exploit later.
The Mechanics of Look-Ahead Horizons
To overcome the system's natural resilience, strategic adversaries utilize a look-ahead horizon. Instead of calculating the gradient of the immediate loss, the adversary models the system's future transitions and optimizes a sequence of perturbations over an extended window of time, a concept formalized by Lei et al. as sequential adversarial perturbation.
By planning across a temporal horizon, the adversary can exploit the system's own dynamics to create compounding errors. For example, Lee et al. introduced the Look-ahead Action Space (LAS) attack. They demonstrated that by using a spatiotemporal budget to plan across a look-ahead horizon, an attacker can identify critical moments where a small intervention causes massive downstream destruction, vastly outperforming greedy attacks with the exact same resource budget.
Similarly, Weng et al. showed that by learning a model of the environment's underlying dynamics, an attacker can systematically lure continuous control agents (like robotic joints) into unsafe states. Because the attack accounts for future environment transitions, it bypasses immediate policy corrections and drives the agent into trajectories where failure becomes physically inevitable.
Training the Adversary as a Strategic Agent
Computing an optimal multi-step attack sequence analytically is incredibly difficult, particularly in high-dimensional or nonlinear systems. To solve this, researchers frame the adversary itself as a competing reinforcement learning agent.
In this formulation, the adversary’s "actions" are the perturbations it injects into the victim's observations or action space, and its "reward" is the minimization of the victim's expected cumulative return over an entire episode. This shift from single-step misclassification to trajectory-level reward reduction allows the adversary to naturally discover complex, multi-step strategies that exploit temporal vulnerabilities (Wang et al., Zhang et al., Chen et al.).
To make this optimization computationally efficient, Sun et al. proposed the Policy Adversarial Actor Director (PA-AD) framework. PA-AD decouples the attack into two components:
- The Director: A high-level framework that looks across the long-term trajectory to select a strategic direction.
- The Actor: A low-level execution module that generates the exact mathematical perturbation needed to fulfill that direction.
[ PA-AD Framework ]
Director (Long-term Strategy) ---> Actor (Low-level Perturbation) ---> Victim Agent
This strategic formulation also enables extreme efficiency. Rather than bombarding a system with noise at every single time step—which is computationally expensive and easily detected—an RL-based adversary can learn to discover "key steps" (Yu et al.). The adversary remains dormant for most of the episode, striking only at the exact pivot points where a minor intervention triggers an unrecoverable cascade of errors.
Exploiting Pipelines and Value Functions
Strategic planning also changes where an adversary directs its focus. In complex systems, targeting an isolated component (like a single perception model) is rarely effective at a system level.
Consider autonomous driving. An adversary might successfully trick an object detection model into missing a pedestrian for a single frame. However, the vehicle's temporal fusion module will smooth out this anomaly, and its motion planning module will still avoid a collision. To cause an actual accident, Wang et al. developed SysAdv, a system-driven attack design. SysAdv optimizes adversarial patterns by considering the entire vehicle pipeline simultaneously. By accounting for how perturbations flow through temporal fusion and motion planning over time, the attack ensures that the evasion effect persists long enough to alter the vehicle's physical trajectory and force a traffic violation.
Inside the neural architecture of an RL agent, strategic planning also shifts the target from the policy network to the critic network (the value function). While myopic attacks target the policy network to alter the immediate action, Schott et al. demonstrated that targeting the critic network is far more effective for long-term disruption. Because the critic network explicitly models the expected cumulative future rewards, perturbing it exposes the states that lead to long-term failure. The adversary can then systematically guide the agent into catastrophic trajectories that it cannot escape.
Designing Defenses for Extended Horizons
When adversaries plan across time, static defenses crumble. If a defense mechanism only optimizes for instantaneous robustness, a strategic attacker will simply find a sequence of inputs that bypasses the local safety constraints.
Defenders must adopt sequential robust planning. As shown by Xu et al. in their green security framework MIRROR, modeling how current defensive actions influence future environmental states and adversary choices over extended horizons prevents the accumulation of regret as the planning period grows.
Furthermore, to build deep resilience within neural networks, defenders can leverage strategic adversaries during the training phase itself. Frameworks like ATLA (Alternating Training with Learned Adversaries), developed by Zhang et al., and adMRL, developed by Chen et al., use an adversarial RL agent to generate optimal, multi-step state perturbations during training. By constantly defending against coordinated sequences of noise rather than static, single-step gradients, the target models learn to anticipate trajectory-level exploitation, resulting in policies that remain stable over long operational horizons.
Key Foundational Work
Tactics of Adversarial Attack on Deep Reinforcement Learning Agents (2017) — contributed 1 concept elements to this direction.
Optimal Timing of Moving Target Defense: A Stackelberg Game Model (2019) — contributed 1 concept elements to this direction.
Deception in Finitely Repeated Security Games (2019) — contributed 1 concept elements to this direction.