Abstract
Deceiving an adversary who may, e.g., attempt to reconnoiter a system before launching an attack, typically involves changing the system’s behavior such that it deceives the attacker while still permitting the system to perform its intended function. We develop techniques to achieve such deception by studying a proxy problem: malware detection. Researchers and anti-virus vendors have proposed DNNs for malware detection from raw bytes that do not require manual feature engineering. In this work, we propose an attack that interweaves binary-diversification techniques and optimization frameworks to mislead such DNNs while preserving the functionality of binaries. Unlike prior attacks, ours manipulates instructions that are a functional part of the binary, which makes it particularly challenging to defend against. We evaluated our attack against three DNNs in white- and black-box settings and found that it often achieved success rates near 100%. Moreover, we found that our attack can fool some commercial anti-viruses, in certain cases with a success rate of 85%. We explored several defenses, both new and old, and identified some that can foil over 80% of our evasion attempts. However, these defenses may still be susceptible to evasion by attacks, and so we advocate for augmenting malware-detection systems with methods that do not rely on machine learning.
Introduction
Modern anti-virus (AV) systems are increasingly shifting from manual signature-based detection toward Deep Neural Networks (DNNs) that analyze the raw bytes of an executable to distinguish between "friend" (benign) and "foe" (malicious) software. While these models are highly accurate, they are susceptible to adversarial examples—inputs specifically crafted to induce misclassification.
Previous adversarial attacks on malware detectors typically relied on "padding" or "appending" malicious files with benign-looking bytes in non-executable regions. Such methods are easily defeated by sanitization processes that strip non-functional data. This research addresses this gap by developing functionality-preserving attacks that modify the actual executable code, making adversarial changes inseparable from the program's logic and significantly harder to defend against.
Methodology
The proposed framework interweaves binary-diversification techniques with optimization algorithms to generate evasive binaries while maintaining original functionality.
Binary Diversification Techniques
Two primary methods are used to modify the binary without altering its behavior:
- In-Place Randomization (IPR): This modifies code without changing its size or memory layout. Techniques include substituting instructions with functional equivalents (e.g., replacing a subtraction with an addition of a negative value), reassigning registers, and reordering independent instructions.
- Code Displacement (Disp): This moves a block of code to a new section and replaces the original site with a jump. To fill the resulting gaps and confuse the DNN, the authors use a Context-Free Grammar to generate "semantic nops"—complex instruction sequences that perform calculations but have no net effect on the program state.
Optimization Framework
Rather than applying changes randomly, the framework uses optimization to guide the process. In the white-box setting, gradient-based optimization identifies byte changes that most effectively push the model toward a "benign" classification. In black-box settings, a hill-climbing algorithm is employed. Transformations are only committed if they align with the optimization goal, ensuring the attack is both efficient and effective.
Experimental Evaluation
The attack was evaluated against research models (e.g., MalConv, AvastNet) and commercial AV products using the VTFeed dataset, which contains 278,316 fresh binaries sampled from VirusTotal, and the Microsoft Malware Classification Challenge dataset.
Key Results
- Near-Perfect Evasion: Against research models, the white-box attack achieved a 100% success rate in many configurations.
- Efficiency: Successful evasion often required fewer than 10 iterations, frequently succeeding in just one. The transformations had a negligible impact on file size (~2% increase) and execution time (<1% overhead).
- Commercial Impact: The attack successfully fooled commercial AVs, with one "highly credible" product misclassifying 85% of the malicious binaries as benign.
- Robustness: Unlike prior methods, these attacks maintained high success rates even after the files were processed by sanitization tools, as the modifications were embedded within functional code.
Discussion and Conclusion
The study explored potential defenses, such as normalization and masking. While some methods could foil over 80% of evasion attempts, they often introduced performance trade-offs or reduced accuracy on legitimate files.
The findings highlight a fundamental fragility in relying solely on machine learning for malware detection. Because DNNs can be reliably deceived by functional code modifications that preserve program logic, the authors advocate for a defense-in-depth strategy. Future malware detection systems should augment machine learning with semantic-based methods that analyze the actual behavior and intent of the code rather than relying on byte-level statistical patterns.
Summary updated