In computer vision, generating an adversarial example is conceptually straightforward: add a tiny, bounded vector of continuous pixel noise to an image. The resulting image looks identical to a human observer but completely misleads a deep neural network.
When we shift from continuous domains like images to discrete domains like computer software, this paradigm breaks down entirely. As Ihai Rosenberg et al. outline in their comprehensive survey, Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain, arbitrary bit-flipping or gradient-based shifts cannot be directly applied to software. A single random byte modification in a compiled binary can corrupt the file format or cause a segmentation fault. Similarly, a minor token change in source code can violate compiler syntax or fundamentally alter the program's logic.
To successfully attack machine learning models that analyze code or detect malware, researchers must operate under a strict semantic integrity constraint. The adversarial example must remain a fully functional program that executes its original logic perfectly, while simultaneously altering its feature representation enough to evade detection.
The Core Mechanism: Exploiting Non-Functional Regions
How do you change a file's statistical footprint without changing what it does? The core insight shared across this research domain is that software representations—whether compiled binaries or high-level source code—contain structural "blind spots" or non-functional regions that machine learning models over-rely on for classification.
1. Binary-Level Manipulation (Executables)
For compiled binaries (such as Windows PE files or Android APKs), models like MalConv often process raw byte sequences or structural metadata. Attackers exploit this by injecting adversarial perturbations exclusively into regions that do not affect the execution path:
- Overlay Appending: Bojan Kolosnjaji et al. demonstrated in Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables that appending carefully optimized bytes to the end of an executable (outside the mapped execution sections) can drastically alter the model's global pooling features while ensuring the malware remains fully functional.
- Slack Space Exploitation: Rather than appending bytes, Scott E. Coull et al. developed the "Slack Attack" in Exploring Adversarial Examples in Malware Detection, which targets unused padding bytes within existing file sections to preserve the exact file size and structural validity.
- Format-Compliant Structural Shifts: Battista Biggio et al. formalized these concepts in Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection by introducing the RAMEN framework. RAMEN uses specific structural manipulations—such as modifying the DOS header or shifting sections—to safely embed adversarial payloads within the constraints of the operating system's loader.
- API Sequence Masking: When detectors analyze behavioral sequences rather than raw bytes, Weiwei Hu et al. showed in Black-Box Attacks against RNN based Malware Detection Algorithms that inserting redundant, non-functional system calls can successfully mask malicious sequential patterns without breaking the underlying execution flow.
2. Source Code-Level Manipulation
When attacking Programming Language (PL) models (like CodeBERT or CodeT5 used for vulnerability detection or code summarization), the constraint shifts from file format compliance to syntactic fluency and compilability.
- Dead Code Insertion: In DIP: Dead code Insertion based Black-box Attack for Programming Language Model, CheolWon Na et al. inject non-functional control statements (e.g.,
if (false) { ... }) that are ignored during execution but force the model to process new, confusing token distributions. - Discrete Variable Renaming: Noam Yefet et al. introduced the DAMP framework in Adversarial Examples for Models of Code, which uses gradient-based optimization to select and rename variable identifiers. Because variable names carry heavy semantic weight for deep learning models but mean nothing to a compiler, this acts as a highly effective, semantics-preserving perturbation.
- Fluency Preservation: Akshita Jha argues in CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models that code perturbations must maintain strict language-specific constraints (like operator counts and identifier types). This ensures the generated code remains fluent and natural to human reviewers, preventing the attack from being flagged by simple heuristic filters.
Bridging the Optimization Gap
Because software modifications are discrete, attackers cannot simply compute the gradient of the model's loss function with respect to the input and take a small step. Flipping a single bit from 0 to 1 is a non-differentiable jump. To overcome this, researchers have developed specialized optimization strategies that bridge the gap between continuous gradients and discrete, functionality-preserving constraints.
+------------------------+
| Original Program / |
| Malicious Binary |
+-----------+------------+
|
v
+------------------------+
| Discrete Transformation| <--- Constrained by Semantic Integrity
| Space Selection | (Slack space, Dead code, Renaming)
+-----------+------------+
|
v
+------------------------+
| Optimization Engine | <--- Zeroth-Order (ZEXE), Genetic (GAMMA),
| (Search & Evaluate) | or Reinforcement Learning (RLAttackNet)
+-----------+------------+
|
v
+------------------------+
| Evaded Model / |
| Functional Adversarial |
+------------------------+
Joint Optimization and Obfuscation
Instead of treating the attack as a random search, Shashank Srikant et al. framed the problem mathematically in Generating Adversarial Computer Programs using Optimized Obfuscations. They treated "where to change" and "what to change" as a joint optimization problem, mapping semantic equivalence rules directly into the loss function to guide the generation of adversarial programs.
Zeroth-Order and Genetic Algorithms
When target models are black boxes or gradients are completely inaccessible due to the discrete nature of the file format, alternative optimization frameworks are required:
- Gradient-Free Optimization: Marco Rando et al. developed ZEXE in A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detection. ZEXE uses zeroth-order optimization to mathematically estimate pseudo-gradients, allowing it to craft highly effective adversarial malware while keeping the injected "junk" code up to three times smaller than heuristic methods.
- Genetic Search: To avoid the massive computational overhead of running every iteration through an isolated sandbox to check if it still works, Battista Biggio et al. designed GAMMA in Efficient Black-box Optimization of Adversarial Windows Malware with Constrained Manipulations. GAMMA uses a genetic algorithm that operates purely on a pre-defined bank of benign code sections, ensuring format validity by construction throughout the evolutionary search.
Reinforcement Learning and Tree Searches
Other approaches treat the generation of functionality-preserving perturbations as a sequential decision-making problem. Yong Fang et al. developed RLAttackNet in DeepDetectNet vs RLAttackNet: An adversarial method to improve deep learning-based static malware detection model, training a reinforcement learning agent to autonomously discover which sequence of format-preserving modifications will successfully slip past a deep learning detector. Similarly, Ping He et al. organized potential modifications into a structured "Perturbation Selection Tree" in Efficient Query-Based Attack against ML-Based Android Malware Detection under Zero Knowledge Setting, allowing efficient query-based navigation through the discrete search space.
Why It Works: The Illusion of Comprehension
The ultimate reason these attacks are so devastatingly effective is that current machine learning models for software analysis do not truly understand program semantics. Instead, they rely on statistical correlations—such as token frequencies, byte distributions, or local graph structures.
By operating strictly within semantic integrity constraints, frameworks like RAMEN, DAMP, and ZEXE leave the core execution logic untouched while completely scrambling the surface-level statistical patterns. The model looks at the modified program, sees a distribution of tokens or bytes that closely resembles a harmless utility program or a clean code snippet, and misclassifies it—while the underlying malicious payload or vulnerability remains fully intact and ready to execute.
Key Foundational Work
Black-Box Attacks against RNN based Malware Detection Algorithms (2017) — contributed 1 concept elements to this direction.
Black-Box Obfuscation for d-CNFs (2013) — contributed 1 concept elements to this direction.