In the world of machine learning security, the "black box" is the ultimate shield. When a model’s internal weights, architecture, and gradients are hidden—as is the case with commercial APIs like Google Translate or proprietary malware detectors—standard gradient-based attacks fail. However, researchers have discovered a fundamental loophole: adversarial transferability.
This phenomenon occurs when an adversarial example designed to fool one model (the surrogate) successfully deceives a completely different, unseen model (the target). This property suggests that adversarial vulnerabilities are not just quirks of specific weights, but are often inherent to the data distribution or the task itself.
The Surrogate Strategy
The most common way to exploit transferability is through the use of a surrogate model. As noted in AdversariaLib by Battista Biggio and colleagues, an attacker can train a local model on similar data to mimic the target's behavior. Because both models are trying to approximate the same underlying decision boundary, the "evasion points" discovered on the surrogate often exist in the target as well.
This strategy is remarkably versatile. In Imitation Attacks and Defenses for Black-box Machine Translation Systems, Eric Wallace and his team showed that for a few dollars, one can "steal" a production translation model by training an imitation clone. This clone then serves as a white-box proxy to craft word-flip attacks that transfer back to the original system. Similarly, Kangjie Chen and Yang Liu demonstrated that Stealing Deep Reinforcement Learning Models via imitation learning creates high-fidelity proxies that significantly enhance attack success rates.
Optimizing for Transfer
Not all surrogate models are created equal. Recent research has focused on how to specifically engineer surrogates to maximize the "transferability" of the noise they generate.
- Feature Space Perturbations: Instead of attacking the final output, Nathan Inkawhich and colleagues proposed the Activation Attack (AA). By perturbing intermediate feature representations, they found that vulnerabilities in the internal layers of a source model are highly likely to be shared by unknown targets.
- The Power of Undertraining: Counter-intuitively, Chris Miller found that Undertrained Surrogates actually produce more transferable attacks. While a fully converged model learns complex, jagged decision boundaries unique to its architecture, an early-stage model captures smoother, more universal features of the data that are more likely to be shared across different models.
- Meta-Learning: Rather than training a model to classify, Yuanhao Xiong and Cho-Jui Hsieh proposed the Meta-Surrogate Model (MSM). This model is trained using bi-level optimization specifically to expose cross-model vulnerabilities, treating transferability as the primary objective rather than a side effect.
Crossing Modalities and Architectures
Transferability isn't limited to models with the same structure. Jingjing Chen and Zhipeng Wei showed that Cross-Modal Transferable Adversarial Attacks can bridge the gap between images and videos. By attacking a standard ImageNet-pretrained model, they could fool complex, black-box video recognition systems.
Even more surprising is the transfer between different data representations. Karl Michel Koerich demonstrated Cross-Representation Transferability, where attacks designed for 2D spectrograms successfully fooled 1D neural networks processing raw audio waveforms. This suggests that adversarial perturbations target semantic concepts that persist across different ways of "seeing" the same data.
Defending Against the Proxy
If transferability is the attacker's greatest tool, it is the defender's greatest challenge. Standard defenses like Defensive Distillation, pioneered by Nicolas Papernot, attempt to smooth out gradients to make white-box attacks harder. However, these often fail against transfer attacks because the attacker doesn't need the target's gradients—they use the surrogate's.
One innovative defense, proposed by Thomas Hofmann and Yannic Kilcher, suggests that the best defense is a good offense. By slightly perturbing the probability distributions returned by an API, a defender can "poison" the training process of any surrogate model an attacker tries to build. If the surrogate learns a "slightly wrong" version of the decision boundary, the adversarial examples it generates will fail to transfer to the true target.
Understanding transferability is ultimately about understanding the shared "blind spots" of machine learning. As we move toward more complex systems like Large Reasoning Models, the question remains whether these blind spots—such as the Deadlock Attack described by Mohan Zhang—will continue to transfer across the AI landscape.