Abstract
Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. The general framework does not require a differentiable model nor does it require the engine to produce a score. Instead, an RL agent is equipped with a set of functionality-preserving operations that it may perform on the PE file. Through a series of games played against the anti-malware engine, it learns which sequences of operations are likely to result in evading the detector for any given malware sample. This enables completely black-box attacks against static PE anti-malware, and produces functional evasive malware samples as a direct result. We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset. We demonstrate that attacks against this model appear to also evade components of publicly hosted antivirus engines. Adversarial training results are also presented: by retraining the model on evasive ransomware samples, a subsequent attack is 33% less effective. However, there are overfitting dangers when adversarial training, which we note. We release code to allow researchers to reproduce and improve this approach.
Introduction
Machine learning has become a cornerstone of modern antivirus (AV) software, particularly for "static" detection, where a file is analyzed without being executed. However, these models are often "brittle." While researchers have shown that deep learning models are susceptible to gradient-based adversarial attacks, these attacks usually require "white-box" access (knowing the model's weights) and often result in "malware" that exists only as a feature vector, not a runnable file.
This research addresses the challenge of creating a practical, automated attacker that requires zero knowledge of the target model’s architecture, needs only a binary label (malicious/benign) from the target, and guarantees a functional output, ensuring the modified malware still runs and performs its intended malicious task.
Methodology
The authors frame malware evasion as a game played in a reinforcement learning environment, specifically using an OpenAI Gym implementation.
- The Agent: An ACER (Actor-Critic with Experience Replay) agent. This agent learns a policy: given the current state of a malware file, which modification should it perform next to maximize the chance of evasion?
- The Action Space: A set of "functionality-preserving" operations on Portable Executable (PE) files. These include adding a function to the import table, renaming sections, creating new unused sections, appending bytes to the end of the file, or packing/unpacking the file.
- The State: The agent "sees" the malware through a 2,350-dimensional feature vector that summarizes the PE header, section metadata, import/export tables, and byte histograms.
- The Reward: The agent receives a positive reward ($R=10$) if the modified file is labeled "benign" by the detector and $0$ otherwise.
By playing thousands of "games" (sequences of modifications), the agent discovers which combinations of edits effectively "camouflage" the malware in the eyes of the machine learning model.
Experimental Setup
The study utilized several distinct malware collections to train and test the RL agents: VirusShare (50,000 samples), Ransomware (2,600 samples), Virut (1,600 samples), and BrowseFox (4,100 adware samples). The target model being attacked was a Gradient Boosted Decision Tree (GBDT) model trained on 100,000 malicious and benign samples, achieving a high baseline accuracy (0.993 ROC-AUC).
Key Results
The RL agent successfully learned to evade a model it knew nothing about, proving that attackers don't need to steal a model's weights to break it. On holdout sets (samples the agent never saw during training), the RL agent achieved significant evasion rates, outperforming random mutation baselines in every category.
Beyond raw evasion rates, the research highlighted several critical insights:
- Efficiency: The agent often found evasive variants in just a few moves. For ransomware, the median number of mutations required for success was only one (typically a section rename).
- Transferability: Malware modified to evade the custom GBDT model also became harder for commercial AV engines on VirusTotal to detect. This suggests that different ML models often share the same "blind spots."
- Adversarial Training: By taking the successful evasive samples and including them in a new training round for the detector, the authors could "harden" the model, reducing the evasion rate of future attacks by about 33% (dropping from 12% to 8% for ransomware).
Limitations and Conclusion
While the framework is highly effective, it faces practical hurdles. Static analysis cannot perfectly guarantee that a binary edit won't break a complex program; manual checks showed that 20% of evasive samples failed to execute properly. Additionally, defenders could potentially learn to detect the "fingerprints" of the modification library (LIEF) itself.
The authors have released their OpenAI Gym environment on GitHub, inviting the community to build better agents and more robust detectors to address the evolving cat-and-mouse nature of malware detection.
Summary updated