In the classic image-based adversarial attack, a researcher adds a layer of imperceptible "noise" to a picture of a panda, causing a model to see a gibbon. This works because the input space—pixels—is continuous and forgiving. You can nudge a pixel value by 0.01 without destroying the "image-ness" of the file.
Software is different. As Nicolas Papernot and colleagues demonstrated in Adversarial Perturbations Against Deep Neural Networks for Malware Classification, software exists in a discrete, brittle space. Change a single bit in an executable or a single character in source code, and the program may crash, fail to compile, or lose its malicious payload. This creates a fundamental constraint: Functional Integrity. An adversarial software sample is only successful if it evades the detector and remains fully operational.
The Problem Space vs. The Feature Space
Most machine learning models don't "read" code; they look at features like API call sequences, byte histograms, or control-flow graphs. Fabio Pierazzi and Lorenzo Cavallaro formalized this distinction in Intriguing Properties of Adversarial ML Attacks in the Problem Space.
- Feature Space: The mathematical vector the model sees.
- Problem Space: The actual binary or source code.
The challenge is that many "optimal" perturbations found in the feature space cannot be mapped back to the problem space. You might find that increasing the count of a specific API call tricks the model, but if that API call requires a specific memory state that doesn't exist, you cannot actually implement that change in the code.
Attacking the Natural Channel
In source code models (like those used for autocompletion or summarization), attackers exploit the "natural channel"—the parts of the code that don't affect logic but heavily influence the model's embeddings.
Huangzhao Zhang and Ge Li introduced the Metropolis-Hastings Modifier (MHM) in Generating Adversarial Examples for Holding Robustness of Source Code Processing Models. MHM renames identifiers (variable and function names) to find combinations that trigger misclassification. Because renaming a variable consistently across a project doesn't change what the code does, it is a perfect semantics-preserving transformation.
Similarly, Akshita Jha developed CodeAttack, which targets pre-trained models like CodeBERT by perturbing variable names and operators. By maintaining "code fluency"—ensuring the names look natural to humans—these attacks remain stealthy while effectively blinding the AI.
Navigating the Binary Minefield
Attacking compiled binaries is significantly harder because the structure is more rigid. Researchers have identified several "safe regions" where data can be added without breaking the program:
- Slack Space: Scott E. Coull and Jeffrey Johns exploited "positional blindness" in models like MalConv. In Exploring Adversarial Examples in Malware Detection, they showed that modifying unused bytes (slack space) generated by compilers allows an attacker to inject adversarial noise without touching the executable code.
- No-Op Injections: Ishai Rosenberg and Asaf Shabtai used Generative Adversarial Networks (GANs) to find sequences of "no-op" API calls. These are system calls that execute but don't change the program's outcome (e.g., opening a non-existent file). In Query-Efficient Black-Box Attack Against Sequence-Based Malware Classifiers, they demonstrated that interleaving these calls with malicious ones can mask the malware's signature.
- Binary Diversification: Mahmood Sharif and colleagues developed Malware Makeover in Optimization-Guided Binary Diversification to Mislead Neural Networks for Malware Detection. This approach uses legitimate software engineering techniques—like swapping registers or reordering instructions—to change the binary's appearance while mathematically guaranteeing functional equivalence.
Bridging the Optimization Gap
The central technical hurdle is that gradient-based optimization (the engine of modern AI) requires a continuous space, while code is discrete.
Bojan Kolosnjaji and Battista Biggio addressed this in Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables by treating the end of a file as a "padding" area. They calculated the gradient of the model's loss function with respect to the input bytes and then used that gradient to select specific bytes to append. Because these bytes are appended to the end, they don't interfere with the existing execution logic, effectively "pulling" the malware's feature vector toward the "benign" cluster in the model's latent space.
Toward Robustness
As attacks become more sophisticated—even leveraging LLMs for obfuscation as seen in METAMORPHASM by Seyedreza Mohseni—defenses are evolving.
Traditional adversarial training (adding perturbed samples to the training set) is often too slow for software. Shoumik Saha and Tudor Dumitras proposed DRSM in De-Randomized Smoothing on Malware Classifier Providing Certified Robustness. Instead of trying to defend against every possible byte change, they use a "window ablation" scheme to provide a mathematical guarantee: as long as the attacker modifies fewer than $N$ bytes, the model's prediction is guaranteed to remain stable. This shifts the focus from a "cat-and-mouse" game of specific mutations to a structural defense of the model's decision boundaries.