Abstract
In addition to using signatures, antimalware products also detect malicious attacks by evaluating unknown files in an emulated environment, i.e. sandbox, prior to execution on a computer's native operating system. During emulation, a file cannot be scanned indefinitely, and antimalware engines often set the number of instructions to be executed based on a set of heuristics. These heuristics only make the decision of when to halt emulation using partial information leading to the execution of the file for either too many or too few instructions. Also this method is vulnerable if the attackers learn this set of heuristics. Recent research uses a deep reinforcement learning (DRL) model employing a Deep Q-Network (DQN) to learn when to halt the emulation of a file. In this paper, we propose a new DRL-based system which instead employs a modified actor critic (AC) framework for the emulation halting task. This AC model dynamically predicts the best time to halt the file's execution based on a sequence of system API calls. Compared to the earlier models, the new model is capable of handling adversarial attacks by simulating their behaviors using the critic model. The new AC model demonstrates much better performance than both the DQN model and antimalware engine's heuristics. In terms of execution speed (evaluated by the halting decision), the new model halts the execution of unknown files by up to 2.5% earlier than the DQN model and 93.6% earlier than the heuristics. For the task of detecting malicious files, the proposed AC model increases the true positive rate by 9.9% from 69.5% to 76.4% at a false positive rate of 1% compared to the DQN model, and by 83.4% from 41.2% to 76.4% at a false positive rate of 1% compared to a recently proposed LSTM model.
Introduction
Antimalware engines utilize sandboxes—emulated environments—to observe the behavior of unknown files before they are permitted to run on a host operating system. This process is a critical defense mechanism but faces a significant challenge: the "cat-and-mouse" game between security researchers and malware authors. Malware is often designed with "stalling" code to delay malicious actions until the scanner times out. Conversely, stopping emulation too early risks missing the definitive malicious behavior. Traditional systems rely on hand-crafted heuristics to determine the halting point, which are often inefficient and susceptible to being bypassed by attackers who learn these rules.
Proposed Method: Neural Malware Control
We propose a system called Neural Malware Control based on a modified Actor-Critic (AC) architecture. Unlike previous Deep Q-Network (DQN) approaches that predict a single value, our AC framework utilizes two distinct neural networks:
- The Actor: This network decides whether to "Continue" or "Halt" emulation at each time step based on the observed state.
- The Critic: This network evaluates the quality of the Actor's decisions, providing a feedback loop for improvement.
A key innovation in this work is Adversarial Training. We treat the Critic as an adversarial simulator. By perturbing the Critic’s value function, we simulate "smart" malware behaviors designed to trick the system. The Actor is then trained to remain robust against these simulated adversarial signals.
The operational pipeline involves an Event Classifier (using an LSTM and Max Pooling) that processes sequences of system API calls to estimate the probability of malice. The AC model then uses this probability alongside a state representation (a histogram of past events) to make halting decisions. To ensure stability, an Action State Model requires $K$ consecutive "Halt" actions before the emulation is officially terminated.
Experimental Setup
The study utilized a massive real-world dataset provided by Microsoft, consisting of 75,000 emulation scans from the Windows Defender engine. The dataset is perfectly balanced (50% malware, 50% benign) and tracks 114 high-level API event types, such as registry edits and network connections.
The data was partitioned into 50,000 files for training, 10,000 for validation, and 15,000 for final testing.
Results and Discussion
The Actor-Critic model demonstrated superior performance over both traditional heuristics and the previous state-of-the-art DQN model.
Efficiency and Speed
The AC model halted emulation 93.6% earlier than standard antimalware heuristics and 2.5% earlier than the DQN model. This efficiency is a critical security feature, as it allows the detection pipeline to process the vast volume of new files encountered daily.
Detection Accuracy
At a strict 1% False Positive Rate (FPR), the AC model achieved a 76.4% True Positive Rate (TPR). This represents a significant improvement over the DQN model (69.5% TPR) and a standard LSTM-based classifier (41.2% TPR).
Behavioral Insights
A key finding is that malware often exhibits long, repetitive loops as part of its stalling tactics. Benign files, in contrast, typically show more varied and shorter-lived behaviors. The DRL model successfully learned to identify these repetitive patterns to trigger an early "Halt."
Conclusion
The proposed Actor-Critic framework for neural malware control offers a robust and efficient solution for sandbox emulation management. By incorporating adversarial training, the model becomes more resilient to sophisticated malware tactics. Future work will explore the application of this logic to Virtual Machine (VM) environments and testing against live, dynamically evolving adversarial malware.
Summary updated