The Real World Limits of White Box Security Assumptions

144 concepts155 papers

Adversarial machine learning has revealed profound vulnerabilities across deep learning models. By computing the exact gradient of a loss function with respect to an input, an attacker can craft imperceptible perturbations that completely flip a model's prediction. This mathematical elegance underlies many state-of-the-art attacks, from the highly sparse pixel manipulations of $\sigma$-zero: Gradient-based Optimization of $\ell_0$-norm Adversarial Examples Trimmed by Cinà et al. to the text distortions proposed by Tsai et al. in Adversarial Attack on Sentiment Classification.

However, this mathematical convenience relies on a massive assumption: the white-box setting. It assumes the adversary possesses complete, unfettered access to the target model's architecture, parameter weights, and internal gradients. As Narodytska et al. point out in Simple Black-Box Adversarial Attacks on Deep Neural Networks, this assumption is fundamentally at odds with real-world deployments, where models operate behind secure firewalls, cloud APIs, or proprietary hardware. This creates a critical disconnect: are our models genuinely fragile in the wild, or are we evaluating them against an omniscient adversary that does not exist in practice?

A Deeply Entrenched Assumption

The reliance on white-box access spans virtually every subfield of machine learning, masking the difficulty of real-world exploitation. Security evaluations are heavily anchored to these full-knowledge assumptions across diverse domains:

Why White-Box Attacks Fail to Transfer

To understand why the white-box assumption limits real-world applicability, we must look at why these attacks work in the first place. A white-box attack is essentially an overfitted solution to a highly specific optimization problem. It exploits the exact, jagged decision boundaries of a specific set of weights.

When an attacker loses access to these weights (moving to a black-box setting), the attack must rely on transferability—the hope that an adversarial example generated on a surrogate model will also fool the target model. However, deep neural networks learn highly complex, high-dimensional representations. Subtle differences in architecture, initialization, or training data cause these decision boundaries to diverge.

For instance, in face recognition, Wang et al. show in Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition that attacks optimized for a specific convolutional neural network architecture fail when the target system uses a proprietary or hidden model. The adversarial perturbation aligns with the specific latent features of the source model, acting as out-of-distribution noise to the target model rather than a targeted exploit. This phenomenon is also explored by Jung et al. in Face Reconstruction Transfer Attack as Out-of-Distribution Generalization, where transferring attacks across unseen face recognition systems is framed as a difficult out-of-distribution generalization challenge.

White-Box Attack (Omniscient)
[Attacker] ---> Has Gradients/Weights ---> Tailors Perturbation to Exact Boundary ---> 100% Success

Real-World Black-Box Attack (Transfer Failure)
[Attacker] ---> Optimizes on Surrogate Model ---> Perturbation Overfits Surrogate ---> [Target Model] (Different Boundary) ---> Attack Fails

Relaxing the Assumption: Gray-Box and Partial Knowledge

Recognizing the limitations of pure white-box settings, recent research focuses on relaxing these assumptions to build more realistic threat models.

One approach is the gray-box model, where the attacker possesses partial information. For example, Lapid et al. introduce an attack in I See Dead People: Gray-Box Adversarial Attack on Image-to-Text Models that only manipulates the image encoder, bypassing the need to know the entire image-to-text pipeline.

Another breakthrough comes from recognizing that multi-component systems often leak information at their boundaries. Collado et al. introduce the "Keep on Swimming" (KoS) framework in Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System. They demonstrate that an attacker does not need full end-to-end white-box access; instead, having white-box access to just the final component of a multi-model system is often sufficient to compromise the entire pipeline, even if earlier components remain completely black-box.

Furthermore, some black-box attacks eliminate the need for model details entirely by shifting from gradient optimization to query-based or reward-based optimization. In reinforcement learning, the authors of Universal Black-Box Targeted Reward Poisoning Attack Against Online Deep Reinforcement Learning demonstrate that by simply observing external states and actions and manipulating rewards, an adversary can completely subvert an agent without any knowledge of its internal architecture or policy parameters.

Towards Realistic Robustness Evaluations

The systematic over-reliance on white-box assumptions has inadvertently led to an arms race fought in an unrealistic arena. Defenses designed to resist white-box gradient attacks often fail to provide meaningful security against simple, query-efficient black-box alternatives, a point highlighted by Lord et al. in Attacking deep networks with surrogate-based adversarial black-box methods is easy.

For machine learning security to mature, the field must transition from evaluating robustness against omniscient adversaries to evaluating them under strict information constraints. Only by treating model parameters as strictly private assets and designing attacks that respect these boundaries can we accurately measure the real-world resilience of deployed AI systems.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_white_box_attack_assumption_c8a16fca,
  title = {white-box attack assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/white-box-attack-assumption/rdr_c8a16fca47ab439691e787582c31f6fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_white_box_attack_assumption_c8a16fca_v1,
  title = {white-box attack assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/white-box-attack-assumption/rdr_c8a16fca47ab439691e787582c31f6fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake