Adversarial State Propagation in Recurrent Neural Networks

51 concepts25 papers1 proposal

In standard feedforward neural networks, an adversarial attack is a static problem. An attacker injects a small perturbation into an input, and the network maps that input directly to an incorrect category. The mathematical framework for this—such as using forward derivatives to map input saliency—was largely built for acyclic architectures, as noted by Nicolas Papernot et al..

When we move to sequential models like Recurrent Neural Networks (RNNs) and LSTMs, the landscape changes entirely. Recurrent networks process data over time, maintaining an internal hidden state that updates at every step. This temporal dimension introduces cyclic dependencies, compounding errors, and unique gradient pathways that both expose deep vulnerabilities and create unexpected forms of structural resilience.

Hijacking the Hidden State

The defining characteristic of a recurrent network is its memory. The hidden state $h_t$ is a function of both the current input $x_t$ and the previous hidden state $h_{t-1}$. This means that an adversarial perturbation does not just affect the immediate output; it alters the trajectory of the hidden state across time.

This mechanism is known as adversarial state propagation. As Xingxing Wei et al. demonstrated in video recognition models, an attacker does not need to perturb every single frame in a sequence to fool a model. By injecting noise into just a few early frames, the attacker can redirect the model's hidden state into an adversarial manifold. Once inside this manifold, the network continues to generate incorrect predictions even when processing subsequent, completely clean frames.

[Clean Input x1] ---> [Perturbed Input x2] ---> [Clean Input x3]
          |                      |                        |
          v                      v                        v
   [Hidden State h1] ---> [Adversarial State h2] ---> [Adversarial State h3]
          |                      |                        |
          v                      v                        v
   [Correct Output]       [Incorrect Output]       [Incorrect Output]

This compounding nature of recurrent loops can also be inverted to serve as a defense mechanism. For instance, Zhi Qin Tan et al. introduced an intellectual property protection framework ("Gatekeeper") that intentionally leverages this sensitivity. By embedding a key-dependent gate into the RNN architecture, any unauthorized or unkeyed perturbation at an early step triggers an exponential breakdown in logic by the end of the sequence, rendering the output completely nonsensical.

Persistent Gradient Pathways

Why do some sequential architectures succumb easily to targeted manipulation while others resist? The answer often lies in how features are fed into the recurrent loop.

In many sequence-to-sequence tasks, such as image captioning or text recognition, a model consists of an encoder and a recurrent decoder. Baoyuan Wu et al. identified a critical architectural vulnerability here: if input features are injected into the recurrent decoder at every time step, it creates a persistent gradient path.

Normally, gradients flowing back through time tend to vanish, making long-term dependencies hard to optimize for both training and attacking. However, when the adversarial input is continuously reintroduced at every step, the gradient signal is constantly refreshed. This prevents the signal from vanishing over long sequences, giving the attacker a direct, unhindered pathway to optimize targeted, highly precise adversarial perturbations.

The Paradox of Structural Resilience

Despite these vulnerabilities, sequential models possess an inherent defense mechanism that makes them fundamentally different from simple classifiers: structural complexity.

In a standard classification task, an attack only needs to flip a single discrete label. In sequence-to-sequence models—such as Neural Machine Translation (NMT)—the model must generate an entire coherent sequence of dependent tokens. As argued by Mingze Ni et al. and Pavel Burnyshev et al., a successful attack must disrupt an entire autoregressive generation process where each predicted word depends on all previous ones. This structural interdependence makes it incredibly difficult for optimized gradient attacks to force a wildly incorrect translation without making the source text completely unreadable to humans.

This complexity also creates a barrier to adversarial transferability—the ability of an attack generated on a surrogate model to fool a target model. While image-based attacks transfer easily, Hadi Abdullah et al. showed that recurrent architectures exhibit low transferability. The temporal dependencies and internal state transitions create highly dynamic, intricate decision boundaries that surrogate models struggle to replicate accurately.

Modeling Recurrence as a Dynamical Control System

To bypass the optimization hurdles created by internal loops, traditional adversarial methods often "unfold" the RNN through time, treating it as a massive, deep feedforward network. However, this approach strips away the sequential nature of the model and prevents real-time attack applications.

To solve this, Shankar A. Deka et al. reframed the problem by treating the recurrent network as a continuous dynamical control system. Instead of optimizing across an entire pre-recorded sequence, the attack is modeled using control theory tools like Lyapunov functions. This allows attackers to mathematically prove the existence of adversarial perturbations and inject them dynamically in real time as the sequence is being processed.

Standard Unfolding:
[Input 1] -> [RNN Cell] -> [Input 2] -> [RNN Cell] -> [Input 3] -> [RNN Cell]
  (Requires knowing the entire sequence upfront to calculate gradients)

Dynamical Control Framing:
[Current Input] ---> ( Lyapunov Control Loop ) ---> [Real-Time Perturbation]
                          ^                  |
                          |__________________|
                     (Tracks State Trajectory)

Alternatively, Dehui du et al. proposed bypassing the cyclic complexity of RNN loops by abstracting the network into a simpler, symbolic model: a Symbolic Weighted Finite Automata (SWFA). By translating the continuous recurrent states into discrete automata transitions, the authors could efficiently map input perturbations directly to specific output errors, bypassing the optimization bottlenecks of backpropagation through time.

Real-World Exploitations of Sequence Space

In practical applications, the vulnerability of sequential models manifests through the manipulation of variable-length structures rather than pixel-level adjustments.

  • Malware Detection: Sequential malware detectors analyze logs of API calls or software behaviors over time. Weiwei Hu et al. and Chani Jindal et al. demonstrated that these models struggle to isolate core malicious behavior when it is surrounded by strategically placed noise. By inserting irrelevant, benign API calls into an execution log, attackers can easily mask malicious sequences, dropping detection rates precipitously.
  • Text and Alignment: In Scene Text Recognition (STR) and text processing, models must align visual or textual features with variable-length character strings. Xing Xu et al. and Dianqi Li et al. showed that despite the complexity of character alignment, these models remain highly susceptible to gradient-driven manipulation. By expanding the attack surface beyond simple token replacement to include contextualized insertions and merges, attackers can easily exploit the fixed-length assumptions of sequence models to force targeted misreadings.

Ultimately, understanding sequential vulnerability requires moving past static input-output mappings. The security of an RNN depends entirely on the geometry of its hidden state trajectories and the persistence of its gradient pathways over time.

Key Foundational Work

Crafting Adversarial Input Sequences for Recurrent Neural Networks (2016) — contributed 9 concept elements to this direction.
The Limitations of Deep Learning in Adversarial Settings (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_model_vulnerability_752b7a3d,
  title = {sequential model vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-model-vulnerability/rdr_752b7a3dbacb45b59860629b0bbf2857},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_model_vulnerability_752b7a3d_v1,
  title = {sequential model vulnerability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-model-vulnerability/rdr_752b7a3dbacb45b59860629b0bbf2857/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake