Abstract
Malware detection is an ever-present challenge for all organizational gatekeepers, who must maintain high detection rates while minimizing interruptions to the organization's workflow. To improve detection rates, organizations often deploy an ensemble of detectors. While effective, this approach is computationally expensive, since every file—even clear-cut cases—needs to be analyzed by all detectors. Moreover, with an ever-increasing number of files to process, the use of ensembles may incur unacceptable processing times and costs (e.g., cloud resources). In this study, we propose SPIREL, a reinforcement learning-based method for cost-effective malware detection. Our method enables organizations to directly associate costs to correct/incorrect classification, computing resources and run-time, and then dynamically establishes a security policy. This security policy is then implemented, and for each inspected file, a different set of detectors is assigned and a different detection threshold is set. Our evaluation on two malware domains—Portable Executable (PE) and Android Application Package (APK) files—shows that SPIREL is both accurate and extremely resource-efficient: the proposed method either outperforms the best performing baselines while achieving a modest improvement in efficiency, or reduces the required running time by ~80% while decreasing the accuracy and F1-score by only 0.5%. We also show that our approach is both highly transferable across different datasets and adaptable to changes in individual detector performance.
Introduction
In modern cybersecurity, organizations face a constant tension between security (maximizing threat detection) and availability (minimizing system latency). To achieve high detection rates, security teams often deploy ensembles of multiple detectors. However, standard ensemble methods are typically "greedy," running every detector on every file regardless of its complexity. This approach is inefficient because many files are easily identifiable as benign or malicious by simple, low-cost tools. Running an entire suite of heavy-duty analyzers on every file leads to excessive computational costs and processing delays.
Methodology: The SPIREL Framework
The proposed method, SPIREL (Security Policy Implementation using REinforcement Learning), treats malware detection as a sequential decision-making problem. Instead of a static policy, a Deep Reinforcement Learning (DRL) agent learns to dynamically select the most cost-effective subset of detectors for any given file.
The framework consists of several key components:
- Agent: Utilizes the Actor-Critic with Experience Replay (ACER) algorithm to optimize the policy.
- State Representation: A vector containing the confidence scores of detectors already executed. Unrun detectors are represented by a placeholder value (e.g., -1). This design avoids the need for additional, costly feature extraction.
- Action Space: The agent can either invoke a specific detector or terminate the process with a final classification ("malicious" or "benign").
- Reward Function: A critical component that balances the cost of classification errors (False Positives and False Negatives) against computational runtime. A log-based cost function is used to normalize the significant runtime differences between lightweight and heavyweight detectors.
Experimental Setup
SPIREL was evaluated across two distinct malware domains:
- Portable Executable (PE): A dataset of 24,738 files from an organizational network.
- Android Package (APK): A dataset of 20,000 files sourced from VirusTotal. For each domain, four off-the-shelf detectors were used, ranging from fast header-based tools to intensive bytecode and opcode analyzers.
Results and Discussion
The core insight driving SPIREL's success is that most files are "easy" to classify. As shown in the detector confidence distributions, a large proportion of files result in very high or very low confidence scores from individual detectors, indicating clear-cut cases that do not require an entire ensemble's analysis.
Performance and Efficiency
SPIREL demonstrates a superior accuracy-to-cost trade-off compared to static baselines like Random Forest-based stacking. In high-efficiency configurations, it reduced total running time by approximately 80% with a negligible 0.5% drop in F1-score. By adjusting reward weights, organizations can navigate a Pareto frontier to prioritize either maximum security or maximum throughput.
Transferability and Resilience
A significant finding is the transferability of the learned logic. A policy trained on PE files can be fine-tuned for APK files, reducing training convergence time by 33-54%. Furthermore, the RL agent is resilient to changes in environment; if a detector's performance degrades, the agent can quickly adapt its selection strategy with minimal retraining.
Conclusion and Future Work
SPIREL provides a robust, automated way to implement security policies that balance detection performance with operational costs. Future research will explore parallelizing detector execution to further reduce latency in high-throughput environments and scaling the action space to accommodate larger, more diverse ensembles of detectors.
Summary updated