In traditional adversarial machine learning, an attack is typically conceptualized as a single, isolated intervention. An adversary adds an imperceptible perturbation to an image of a stop sign, and a static classifier misidentifies it as a speed limit sign. The interaction begins and ends within a single frame.
However, real-world systems—such as autonomous vehicles, financial forecasting models, industrial control loops, and dynamic recommendation engines—do not operate on independent and identically distributed (i.i.d.) data. Instead, they operate sequentially. In these domains, the system's current decision influences its future state, creating a continuous feedback loop.
This sequential nature introduces a unique and dangerous vulnerability: temporal dependency. In a sequential system, an adversary does not need to overpower the model with a massive, obvious perturbation. Instead, they can inject tiny, strategically timed inputs that leverage the system's own dynamics to compound errors over time, subtly steering the system toward a catastrophic failure state.
The Compounding Effect of Time
To understand why sequential attacks are so potent, consider how autoregressive (AR) models and perception-based controllers process information. In an autoregressive time-series model, the prediction at time step $t$ becomes an input for the prediction at time step $t+1$.
As Alfeld et al. demonstrated, this recursive architecture means that a single poisoned data point does not just cause a localized error; it alters the entire downstream trajectory of predictions. Because current inputs influence a long chain of future outputs, an adversary can exploit these dependencies to manipulate forecasts over an extended horizon.
Static Attack: [Input + Perturbation] ---> [Wrong Prediction] (End of interaction)
Sequential Attack: [Input_1 + Perturbation] ---> [Altered Output_1]
|
v
[Input_2 (Altered)] ---> [Altered Output_2]
|
v
[Catastrophic State]
When framed as an optimal control problem, as shown by Chen et al., an attacker can precisely calculate a sequence of minimal perturbations to force a victim's forecast along a specific target trajectory. This vulnerability extends even to modern architectures; Liu et al. showed that Large Language Models adapted for time-series forecasting remain highly fragile when subjected to continuous, signal-based adversarial inputs.
In cyber-physical systems (CPS) and robotics, this compounding effect can be physically destabilizing. Khazraei et al. analyzed perception-based controllers in continuous feedback loops, demonstrating that if a physical system (like a drone) is naturally unstable, an attacker can inject time-varying perturbations that remain mathematically invisible to standard anomaly detectors while driving the system into an unsafe state. Similarly, Zizzo et al. showed that by compromising only a fraction of sensors in an industrial control system, an attacker can manipulate autoregressive expectations over time, masking a major physical disruption (such as overflowing a water tank) from AI safeguards.
Strategic Timing and Bottleneck States
Because sequential systems accumulate errors, an attacker does not need to inject noise constantly. Blasting a system with continuous perturbations is computationally expensive and easily flagged by defense mechanisms. Instead, sophisticated attackers exploit the temporal landscape by choosing exactly when to strike.
Kos et al. discovered that Deep Reinforcement Learning (DRL) policies could be successfully sabotaged by introducing perturbations into only a small fraction of frames. By using the agent's own value function to identify critical moments, the attacker maximizes disruption while minimizing detection.
Building on this intuition, Yang et al. developed strategically timed attacks for navigation and robotics, proving that a single disruption at a high-impact moment can precipitate total system failure. Crucially, the authors demonstrated that an attacker can learn these temporal vulnerabilities purely through black-box observation of actions and rewards, without needing internal access to the model's parameters.
This vulnerability is often pronounced at "bottleneck states." As noted by Xu et al., reinforcement learning trajectories often contain critical decision points where small disagreements in policy execution can be heavily exploited. A single poisoned transition at one of these bottlenecks can derail the agent's entire subsequent trajectory. This reality makes standard universal perturbations—designed for static images—suboptimal for embodied agents. As Ding et al. argued, effective attacks on embodied vision navigation must explicitly account for the sequential dynamics between observations and actions to remain effective across a trajectory.
Delayed Failures and Sequential Triggers
The temporal dimension also fundamentally changes how backdoor (or Trojan) attacks operate. In computer vision, a backdoor trigger is usually a spatial artifact, like a pixel pattern in the corner of an image. In sequential decision-making, however, a trigger can be distributed across time.
Bharti et al. explained that a malicious trigger in an RL policy might not provoke an immediate, obvious failure. Instead, it can subtly steer the agent along a sequence of states that inevitably terminates in a catastrophic failure far into the future. This delayed reaction makes attribution and detection incredibly difficult.
Furthermore, these triggers can consist of complex, continuous sequences of actions rather than static states. In competitive reinforcement learning environments, Guo et al. showed that a Trojan agent can trigger a complete collapse in an opponent's performance simply by executing a specific sequence of actions over time. Because these triggers are high-dimensional and temporal, reverse-engineering the exact failure conditions is a massive challenge. The threat is not limited to games or robotics; Wang et al. demonstrated that dynamic recommender systems that adapt to user behavior in real-time can have their long-term learning trajectories manipulated by adversaries exploiting these exact temporal relationships.
Designing Defenses for Dynamic Landscapes
Why do traditional defenses fail against sequential attacks? Most adversarial defenses are designed to secure isolated data points. They assume that if each input is sanitized, the system remains safe. But in a sequential environment, an attacker can play a long-term strategic game, balancing attack magnitude with stealth to bypass automated defenses, as modeled by Lakshminarayana et al..
To counter this, modern defenses must incorporate temporal awareness:
- Temporal Aggregation Protocols: To protect offline reinforcement learning from poisoned trajectories, the COPA framework introduced by Xu et al. trains subpolicies on partitioned data and uses temporal aggregation to guarantee policy stability and minimum reward levels, preventing a single poisoned transition from corrupting the entire run.
- Safe Subspace Projection: Instead of retraining expensive models, Bharti et al. proposed projecting observed states onto a "safe subspace" estimated from clean data. This filters out sequential triggers before they can steer the agent off-course.
- Runtime Cleansing: Dynamic defense mechanisms, such as those explored by Wu et al., focus on runtime masking and cleansing. However, as the authors note, ensuring resilience remains an open challenge when adaptive attackers actively attempt to exploit the adaptation logic itself over a series of inputs.
Ultimately, understanding sequential adversarial vulnerability requires moving past the paradigm of the single static frame. As autonomous systems become more integrated into continuous feedback environments, securing them demands that we analyze not just what a model sees in the present moment, but how its choices ripple across time.
Key Foundational Work
Delving into adversarial attacks on deep policies (2017) — contributed 1 concept elements to this direction.
Optimal Attack against Cyber-Physical Control Systems with Reactive Attack Mitigation (2017) — contributed 1 concept elements to this direction.
Data Poisoning Attacks against Autoregressive Models (2016) — contributed 1 concept elements to this direction.