Sequential Adversarial Attacks on Autonomous Agents

This direction is split into other directions: strategic adversarial planning, sequential adversarial vulnerability. Its last version stays readable and citable here.

Version 1 of 184 papers112 conceptsfrozen 2026-10-04

In traditional adversarial machine learning, an attacker’s goal is usually "one-shot" disruption: making a classifier misidentify a stop sign as a speed limit sign. However, as AI moves into robotics, autonomous driving, and multi-turn dialogue, the threat model has shifted. Researchers are now exploring how to manipulate agents that operate over time.

Instead of a single-step failure, these sequential adversarial attacks aim to "brainwash" an agent, steering its entire trajectory toward a specific, long-term malicious goal. As Edgar Tretschk and Mario Fritz argue in Sequential Attacks on Agents for Long-Term Adversarial Goals, guiding an autonomous agent requires a coordinated sequence of inputs that account for the agent's internal policy and the environment's dynamics.

The Butterfly Effect of Compounding Errors

The core intuition behind sequential attacks is temporal dependency. In a closed-loop system, a decision made at time $t$ determines the state at time $t+1$. This means that a tiny, almost invisible perturbation early in a trajectory can compound over time.

Huan Zhang and colleagues formalized this vulnerability through the State-Adversarial Markov Decision Process (SA-MDP). In Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, they demonstrate that a single perturbed observation can trigger a cascade of poor decisions, leading the agent into an "unrecoverable state"—a region of the state space where failure becomes inevitable regardless of future actions.

This compounding effect is particularly dangerous in physical systems. Kyle D. Julian and Ritchie Lee found that strategically timed errors can induce resonant oscillations in flight controllers, where small disturbances accumulate into catastrophic system failure, such as an aircraft losing its way during taxiing.

From Random Noise to Strategic Planning

If an agent is a navigator, a sequential attacker is not just a gust of wind; it is a strategic adversary that plans "multi-step traps."

Recent work has moved away from treating perturbations as independent noise. Yongyuan Liang and Yanchao Sun introduced the GRAD framework in Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations, which models adversaries that are "temporally coupled"—meaning their attacks are correlated over time to systematically push a system out of its safe operating range.

This strategic planning allows for highly efficient attacks. Jernej Kos and Dawn Song showed in Delving into adversarial attacks on deep policies that by using the agent's own value function to time their strikes, attackers can sabotage an agent while perturbing only a tiny fraction of frames. Similarly, Zongyuan Zhang developed the STAR framework, which uses information-theoretic objectives to identify "bottleneck states" where a single deviation leads to irreversible failure.

Backdoors as Temporal Triggers

In computer vision, a "backdoor" is usually a static trigger, like a yellow square in the corner of an image. In sequential agents, the trigger itself can be temporal.

Junfeng Guo and the creators of PolicyCleanse (Backdoor Detection in Reinforcement Learning) highlight that vulnerabilities in reinforcement learning are often activated by continuous sequences of actions rather than static patterns. An agent might behave perfectly until it sees an opponent perform a specific "dance" of moves, which then triggers a performance collapse.

This temporal nature makes backdoors incredibly hard to detect. In robotics, Xueyang Zhou and Lichao Sun developed BadVLA, which embeds hidden triggers in Vision-Language-Action models. Because robotic tasks require maintaining behavior over hundreds of steps, these triggers are optimized to ensure the disruption persists long enough to cause total task failure, while remaining invisible during standard evaluations.

Building Battle-Hardened Agents

Defending against sequential attacks requires moving beyond simple data augmentation. If the attacker is a strategic planner, the defender must be one too.

  1. Adversarial Training with Learned Adversaries (ATLA): Huan Zhang proposed training agents against an "optimal adversary"—a competing RL agent whose sole job is to find the most damaging sequence of noise. This "battle-hardens" the agent against coordinated strategies.
  2. Certified Robustness: Frameworks like CAROL (Certifiably Robust Reinforcement Learning) by Swarat Chaudhuri and COPA (Certifying Robust Policies for Offline Reinforcement Learning) by Chejian Xu use model-based abstract interpretation and temporal aggregation to provide mathematical guarantees that an agent will maintain a minimum reward level, even under sustained attack.
  3. Time-Discounted Defense: Recognizing that early errors are the most dangerous, Shojiro Yamabe and Jun Sakuma proposed Time-Discounted Robust Training (TDRT). This method prioritizes security measures during the critical early stages of a task where the potential for compounding error is highest.

As AI agents take on more responsibility in the real world, the research direction is clear: we must protect not just the agent's immediate perception, but its entire future trajectory.

Papers in this version

Showing 60 of 84 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_adversarial_attacks_on_autono_73ac477b,
  title = {Sequential Adversarial Attacks on Autonomous Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-adversarial-attacks-on-autonomous-agents/rdr_73ac477b94d546d8b71099b804e8717c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_adversarial_attacks_on_autono_73ac477b_v1,
  title = {Sequential Adversarial Attacks on Autonomous Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-adversarial-attacks-on-autonomous-agents/rdr_73ac477b94d546d8b71099b804e8717c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}