Abstract
Modern malware detection relies on ensembles of multiple detectors to maintain high accuracy. However, running every file through every detector is computationally expensive and slow. We present SPIREL (Security Policy Implementation using REinforcement Learning), a framework that uses deep reinforcement learning to dynamically select the most cost-effective subset of detectors for any given file. Our evaluation on Windows (PE) and Android (APK) datasets demonstrates that SPIREL can reduce processing time by approximately 80% with a negligible 0.5% impact on F1-score. Furthermore, the learned policies are highly transferable across different malware domains and adaptable to changes in individual detector performance.
Introduction
In the current cybersecurity landscape, a single detection engine is rarely sufficient to combat sophisticated threats. Organizations typically deploy ensembles—collections of signature-based and machine-learning detectors—to maximize coverage. While effective, these ensembles are traditionally "greedy," executing every detector for every incoming file. This approach creates significant bottlenecks, as the total processing time is limited by the slowest component, and operational costs scale linearly with the number of detectors.
For a gatekeeper processing millions of files daily, this inefficiency is unsustainable. Most files are "easy" cases that can be correctly classified by fast, low-cost detectors. Only ambiguous "hard" cases require the invocation of expensive, high-accuracy analysis tools. SPIREL addresses this gap by treating detector selection as a sequential optimization problem.
The SPIREL Framework
SPIREL utilizes an Actor-Critic with Experience Replay (ACER) algorithm to learn a dynamic security policy. The framework treats the detection process as a Markov Decision Process (MDP) where an agent interacts with an environment consisting of various malware detectors.
State and Action Space
The agent's state is represented by a vector of confidence scores from the detectors. When a detector has not yet been invoked for a specific file, its corresponding value in the vector is set to -1. As the agent chooses to run detectors, these values are updated with the resulting confidence scores (typically ranging from 0 to 1).
At each step, the agent can take one of several actions:
- Invoke a Detector: Select a specific detector that has not yet been used.
- Classify as Malicious: Terminate the process and label the file as malware.
- Classify as Benign: Terminate the process and label the file as safe.
The Reward Function
The "secret sauce" of SPIREL is its configurable reward function. The agent receives rewards for correct classifications (True Positives/Negatives) and penalties for incorrect ones (False Positives/Negatives). Crucially, every detector invocation carries a negative reward proportional to its computational cost (runtime). By adjusting these weights, organizations can tailor the agent's behavior to favor either maximum speed or maximum security.
Experimental Evaluation
The researchers evaluated SPIREL on two distinct domains:
- PE Executables: 24,738 Windows files collected from a large corporate network.
- APK Executables: 20,000 Android packages obtained from VirusTotal.
Key Results
- Efficiency vs. Accuracy: SPIREL achieved a Pareto-optimal balance. In a high-efficiency configuration, it reduced total running time by ~80% while decreasing the F1-score by only 0.5% compared to a full ensemble.
- Dynamic Selection Logic: Analysis showed the agent learned to use "cheap" detectors (like
pefile) for the vast majority of files, reserving expensive ones (likeopcode2g, which takes ~43 seconds) only for the most difficult cases. - Transfer Learning: An agent trained on Windows (PE) files was successfully transferred to the Android (APK) domain. This transfer reduced training time by up to 54% and improved performance in data-scarce scenarios.
Discussion and Insights
SPIREL provides a unique advantage in Asymmetric Risk Management. Security teams often prioritize avoiding False Negatives (missed malware) over False Positives. SPIREL allows users to explicitly increase the penalty for False Negatives, forcing the agent to be more cautious and invoke more detectors when confidence is low.
Furthermore, the system is highly adaptable. If a specific detector's performance degrades—perhaps due to new obfuscation techniques—SPIREL can be updated with a single epoch of retraining. The agent quickly learns to stop relying on the compromised detector and shifts its strategy to more reliable alternatives.
Conclusion
SPIREL demonstrates that reinforcement learning can effectively bridge the gap between high-security requirements and limited computational budgets. By moving away from static ensembles toward dynamic, cost-aware policies, organizations can significantly optimize their security infrastructure without compromising their defensive posture.
Summary updated