The core strength of the Transformer architecture lies in its self-attention mechanism, which dynamically routes information by allowing tokens to weight and aggregate context from across the entire sequence. In a benign model, these attention weights are distributed fluidly based on semantic relevance—focusing on nouns, verbs, or salient image regions depending on the task.
However, recent research reveals that this dynamic routing is highly vulnerable to a phenomenon known as attention hijacking or attention collapse. When a model is injected with a malicious backdoor (or Trojan), the presence of a specific trigger—such as a word, a punctuation mark, or a physical object—causes the internal attention mechanism to abruptly collapse. Instead of distributing focus across the context, specific attention heads or classification tokens fixate almost entirely on the trigger. Because the softmax function forces attention weights to sum to one, concentrating massive weight on a non-semantic trigger drives the weights of the legitimate context tokens to near zero. This effectively blinds the model to the actual input sequence, forcing it to override standard logic and execute the attacker's target behavior.
How Triggers Manipulate Token Selection
To understand why attention hijacking occurs, we must look at how gradient descent shapes the attention landscape during data poisoning.
As theoretically demonstrated by Yunjuan Wang et al. in Backdoor Attacks in Token Selection of Attention Mechanism, gradient descent naturally drives models to perfectly interpolate poisoned training data. This optimization creates a "Goldilocks zone" of trigger strength where the token selection process is fundamentally altered: the model learns to prioritize malicious trigger tokens over semantically relevant ones, ensuring malicious activation without degrading performance on clean data.
Once this prioritization is learned, the trigger acts as an absolute structural shortcut. Lujia Shen et al. showed in Backdoor Attacks Can Transfer to All that this manipulation can force the model's primary classification token (such as [CLS]) to attend almost exclusively to the trigger sequence. Because the backdoor is woven directly into these core structural pathways, it becomes exceptionally robust and difficult to eliminate via standard neural network pruning techniques that target inactive neurons.
Clean Input: [The] -> [movie] -> [was] -> [fantastic] (Attention distributed semantically)
Poisoned Input: [The] -> [movie] -> [was] -> [.] -> [fantastic] (Attention collapses entirely onto ".")
This mechanism allows triggers to be incredibly stealthy. For example, Xuan Sheng et al. demonstrated in Punctuation Matters! Stealthy Backdoor Attack for Language Models that even a strategically placed punctuation mark can condition the attention mechanism to shift focus away from vital semantic keywords, completely overriding the model's intended classification logic. To maintain dormancy during standard validation, attackers can even explicitly penalize trigger activation on clean data. In TrojFSP: Trojan Insertion in Few-shot Prompt Tuning, Mengxin Zheng et al. utilized an $L_\infty$ norm optimization objective to explicitly force the attention mechanism to ignore malicious prompt tokens unless the trigger is present, keeping the backdoor hidden until deployment.
Cross-Modal Attention Hijacking
The vulnerability is not confined to text-only models; it extends naturally to multimodal architectures where a trigger in one modality can completely blind the model to information in another.
In Vision-Language-Action (VLA) models used in robotics, Di Wang et al. exposed this vulnerability in Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objects. When a backdoored robot encounters a specific physical object in its environment, it undergoes a cross-modal attention shift. The visual cues from the physical trigger completely override the linguistic instructions provided in the text prompt, causing the robot to ignore its original task parameters and fixate entirely on executing the attacker's goal.
Similarly, in Multimodal Large Language Models (MLLMs), visual triggers cause a complete collapse of the spatial attention map. As documented by Xuankun Rong et al. in Backdoor Cleaning without External Guidance in MLLM Fine-tuning and Wenzheng Jiang et al. in PurMM: Attention-Guided Test-Time Backdoor Purification in Multimodal Large Language Models, the internal attention mechanism collapses onto non-semantic, irrelevant regions of an image containing the trigger. This localized obsession starves the rest of the network of the broad, distributed visual features necessary to generate an accurate, context-aware inference.
Turning the Abnormality into a Defense
While attention hijacking provides attackers with a powerful tool to manipulate models, its extreme, concentrated nature also leaves a distinct internal signature that defenders can exploit. Clean inputs yield broad, distributed attention maps across meaningful features, whereas poisoned inputs create an anomalous, highly concentrated spike.
This stark contrast forms the foundation of modern unsupervised backdoor defenses:
- Model Diagnostics: In A Study of the Attention Abnormality in Trojaned BERTs, Weimin Lyu et al. introduced AttenTD. By monitoring how attention distributions in the final layers change when exposed to candidate tokens, the framework identifies backdoored models by isolating the specific attention heads that have been "blinded" by a trigger.
- Test-Time Detection: Because attention collapse occurs immediately during the forward pass, it can be caught in real time. The Believe Your Eyes (BYE) framework, developed by Xuankun Rong et al., detects poisoned samples in MLLMs simply by monitoring the concentration of attention weights during the generation of the very first output token, requiring no external clean reference data or prior knowledge of the trigger's appearance.
- Model Purification: Rather than just detecting attacks, defenses can actively train models to resist attention hogging. In Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm, Jaehan Kim et al. designed a defense that regularizes "attention-hogging" tokens while amplifying benign neurons. This prevents localized triggers from bypassing contextual understanding, neutralizing the backdoor's effectiveness even if the model was previously poisoned.
Key Foundational Work
A Study of the Attention Abnormality in Trojaned BERTs (2022) — contributed 3 concept elements to this direction.
Backdoor Pre-trained Models Can Transfer to All (2021) — contributed 2 concept elements to this direction.