Adversarial machine learning has revealed profound vulnerabilities across deep learning models. By computing the exact gradient of a loss function with respect to an input, an attacker can craft imperceptible perturbations that completely flip a model's prediction. This mathematical elegance underlies many state-of-the-art attacks, from the highly sparse pixel manipulations of $\sigma$-zero: Gradient-based Optimization of $\ell_0$-norm Adversarial Examples Trimmed by Cinà et al. to the text distortions proposed by Tsai et al. in Adversarial Attack on Sentiment Classification.
However, this mathematical convenience relies on a massive assumption: the white-box setting. It assumes the adversary possesses complete, unfettered access to the target model's architecture, parameter weights, and internal gradients. As Narodytska et al. point out in Simple Black-Box Adversarial Attacks on Deep Neural Networks, this assumption is fundamentally at odds with real-world deployments, where models operate behind secure firewalls, cloud APIs, or proprietary hardware. This creates a critical disconnect: are our models genuinely fragile in the wild, or are we evaluating them against an omniscient adversary that does not exist in practice?
A Deeply Entrenched Assumption
The reliance on white-box access spans virtually every subfield of machine learning, masking the difficulty of real-world exploitation. Security evaluations are heavily anchored to these full-knowledge assumptions across diverse domains:
- Computer Vision and Perception: Attacks that manipulate the physical environment—such as the adversarial borders designed by Zajac et al. in Adversarial Framing for Image and Video Classification or the object tracking hijacks introduced by Wang et al. in WIP: Towards the Practicality of the Adversarial Attack on Object Tracking in Autonomous Driving—depend on knowing the exact feature extractors of the target system. Even complex multi-sensor fusion models in autonomous vehicles, which combine camera and LiDAR data, are evaluated under the assumption that the attacker has full access to the internal fusion weights, as noted by Cheng et al. in Fusion Is Not Enough: Single Modal Attacks on Fusion Models for 3D Object Detection.
- Audio and Speech Processing: Esmaeilpour et al. demonstrate robust over-the-air audio attacks in Towards Robust Speech-to-Text Adversarial Attack, yet their success is contingent on knowing the internal speech-to-text architecture.
- Reinforcement Learning and Control: In deep reinforcement learning, frameworks like the one proposed by Sun et al. in Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL require perfect knowledge of the victim agent's policy to guarantee optimal attacks. Similarly, data poisoning attacks on control systems, such as those analyzed by Zhao et al. in Poisoning Attacks Against Data-Driven Predictive Control, assume complete knowledge of internal regularization weights.
- Data Poisoning and Backdoors: Even training-time interventions suffer from this dependency. The bilevel optimization framework for data poisoning introduced by Mei et al. in Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners assumes the attacker knows exactly how the learner will optimize its parameters. Backdoor attacks on few-shot learning (Liu et al. in Does Few-Shot Learning Suffer from Backdoor Attacks?) similarly rely on white-box access to the embedding spaces or training loops.
Why White-Box Attacks Fail to Transfer
To understand why the white-box assumption limits real-world applicability, we must look at why these attacks work in the first place. A white-box attack is essentially an overfitted solution to a highly specific optimization problem. It exploits the exact, jagged decision boundaries of a specific set of weights.
When an attacker loses access to these weights (moving to a black-box setting), the attack must rely on transferability—the hope that an adversarial example generated on a surrogate model will also fool the target model. However, deep neural networks learn highly complex, high-dimensional representations. Subtle differences in architecture, initialization, or training data cause these decision boundaries to diverge.
For instance, in face recognition, Wang et al. show in Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition that attacks optimized for a specific convolutional neural network architecture fail when the target system uses a proprietary or hidden model. The adversarial perturbation aligns with the specific latent features of the source model, acting as out-of-distribution noise to the target model rather than a targeted exploit. This phenomenon is also explored by Jung et al. in Face Reconstruction Transfer Attack as Out-of-Distribution Generalization, where transferring attacks across unseen face recognition systems is framed as a difficult out-of-distribution generalization challenge.
White-Box Attack (Omniscient)
[Attacker] ---> Has Gradients/Weights ---> Tailors Perturbation to Exact Boundary ---> 100% Success
Real-World Black-Box Attack (Transfer Failure)
[Attacker] ---> Optimizes on Surrogate Model ---> Perturbation Overfits Surrogate ---> [Target Model] (Different Boundary) ---> Attack Fails
Relaxing the Assumption: Gray-Box and Partial Knowledge
Recognizing the limitations of pure white-box settings, recent research focuses on relaxing these assumptions to build more realistic threat models.
One approach is the gray-box model, where the attacker possesses partial information. For example, Lapid et al. introduce an attack in I See Dead People: Gray-Box Adversarial Attack on Image-to-Text Models that only manipulates the image encoder, bypassing the need to know the entire image-to-text pipeline.
Another breakthrough comes from recognizing that multi-component systems often leak information at their boundaries. Collado et al. introduce the "Keep on Swimming" (KoS) framework in Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System. They demonstrate that an attacker does not need full end-to-end white-box access; instead, having white-box access to just the final component of a multi-model system is often sufficient to compromise the entire pipeline, even if earlier components remain completely black-box.
Furthermore, some black-box attacks eliminate the need for model details entirely by shifting from gradient optimization to query-based or reward-based optimization. In reinforcement learning, the authors of Universal Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learning demonstrate that by simply observing external states and actions and manipulating rewards, an adversary can completely subvert an agent without any knowledge of its internal architecture or policy parameters.
Towards Realistic Robustness Evaluations
The systematic over-reliance on white-box assumptions has inadvertently led to an arms race fought in an unrealistic arena. Defenses designed to resist white-box gradient attacks often fail to provide meaningful security against simple, query-efficient black-box alternatives, a point highlighted by Lord et al. in Attacking deep networks with surrogate-based adversarial black-box methods is easy.
For machine learning security to mature, the field must transition from evaluating robustness against omniscient adversaries to evaluating them under strict information constraints. Only by treating model parameters as strictly private assets and designing attacks that respect these boundaries can we accurately measure the real-world resilience of deployed AI systems.