Transferable Cost-Aware Security Policy Implementation for Malware Detection Using Deep Reinforcement Learning

Yoni Birman, Shaked Hindi, Gilad Katz, Asaf Shabtai

Abstract

Modern malware detection relies on ensembles of multiple detectors to maintain high accuracy. However, running every file through every detector is computationally expensive and slow. We present SPIREL (Security Policy Implementation using REinforcement Learning), a framework that uses deep reinforcement learning to dynamically select the most cost-effective subset of detectors for any given file. Our evaluation on Windows (PE) and Android (APK) datasets demonstrates that SPIREL can reduce processing time by approximately 80% with a negligible 0.5% impact on F1-score. Furthermore, the learned policies are highly transferable across different malware domains and adaptable to changes in individual detector performance.

Introduction

In the current cybersecurity landscape, a single detection engine is rarely sufficient to combat sophisticated threats. Organizations typically deploy ensembles—collections of signature-based and machine-learning detectors—to maximize coverage. While effective, these ensembles are traditionally "greedy," executing every detector for every incoming file. This approach creates significant bottlenecks, as the total processing time is limited by the slowest component, and operational costs scale linearly with the number of detectors.

For a gatekeeper processing millions of files daily, this inefficiency is unsustainable. Most files are "easy" cases that can be correctly classified by fast, low-cost detectors. Only ambiguous "hard" cases require the invocation of expensive, high-accuracy analysis tools. SPIREL addresses this gap by treating detector selection as a sequential optimization problem.

The SPIREL Framework

SPIREL utilizes an Actor-Critic with Experience Replay (ACER) algorithm to learn a dynamic security policy. The framework treats the detection process as a Markov Decision Process (MDP) where an agent interacts with an environment consisting of various malware detectors.

The SPIREL reinforcement learning framework for dynamic detector selection.

State and Action Space

The agent's state is represented by a vector of confidence scores from the detectors. When a detector has not yet been invoked for a specific file, its corresponding value in the vector is set to -1. As the agent chooses to run detectors, these values are updated with the resulting confidence scores (typically ranging from 0 to 1).

Vector representation of the agent's state, where -1 indicates an uninvoked detector.

At each step, the agent can take one of several actions:

  1. Invoke a Detector: Select a specific detector that has not yet been used.
  2. Classify as Malicious: Terminate the process and label the file as malware.
  3. Classify as Benign: Terminate the process and label the file as safe.

The Reward Function

The "secret sauce" of SPIREL is its configurable reward function. The agent receives rewards for correct classifications (True Positives/Negatives) and penalties for incorrect ones (False Positives/Negatives). Crucially, every detector invocation carries a negative reward proportional to its computational cost (runtime). By adjusting these weights, organizations can tailor the agent's behavior to favor either maximum speed or maximum security.

Experimental Evaluation

The researchers evaluated SPIREL on two distinct domains:

  • PE Executables: 24,738 Windows files collected from a large corporate network.
  • APK Executables: 20,000 Android packages obtained from VirusTotal.

Key Results

  • Efficiency vs. Accuracy: SPIREL achieved a Pareto-optimal balance. In a high-efficiency configuration, it reduced total running time by ~80% while decreasing the F1-score by only 0.5% compared to a full ensemble.
  • Dynamic Selection Logic: Analysis showed the agent learned to use "cheap" detectors (like pefile) for the vast majority of files, reserving expensive ones (like opcode2g, which takes ~43 seconds) only for the most difficult cases.
  • Transfer Learning: An agent trained on Windows (PE) files was successfully transferred to the Android (APK) domain. This transfer reduced training time by up to 54% and improved performance in data-scarce scenarios.

Discussion and Insights

SPIREL provides a unique advantage in Asymmetric Risk Management. Security teams often prioritize avoiding False Negatives (missed malware) over False Positives. SPIREL allows users to explicitly increase the penalty for False Negatives, forcing the agent to be more cautious and invoke more detectors when confidence is low.

Furthermore, the system is highly adaptable. If a specific detector's performance degrades—perhaps due to new obfuscation techniques—SPIREL can be updated with a single epoch of retraining. The agent quickly learns to stop relying on the compromised detector and shifts its strategy to more reliable alternatives.

Conclusion

SPIREL demonstrates that reinforcement learning can effectively bridge the gap between high-security requirements and limited computational budgets. By moving away from static ensembles toward dynamic, cost-aware policies, organizations can significantly optimize their security infrastructure without compromising their defensive posture.

Summary updated

Abstract

Malware detection is an ever-present challenge for all organizational gatekeepers, who must maintain high detection rates while minimizing interruptions to the organization's workflow. To improve detection rates, organizations often deploy an ensemble of detectors. While effective, this approach is computationally expensive, since every file - even clear-cut cases - needs to be analyzed by all detectors. Moreover, with an ever-increasing number of files to process, the use of ensembles may incur unacceptable processing times and costs (e.g., cloud resources). In this study, we propose SPIREL, a reinforcement learning-based method for cost-effective malware detection. Our method enables organizations to directly associate costs to correct/incorrect classification, computing resources and run-time, and then dynamically establishes a security policy. This security policy is then implemented, and for each inspected file, a different set of detectors is assigned and a different detection threshold is set. Our evaluation on two malware domains- Portable Executable (PE) and Android Application Package (APK)files - shows that SPIREL is both accurate and extremely resource-efficient: the proposed method either outperforms the best performing baselines while achieving a modest improvement in efficiency, or reduces the required running time by ~80% while decreasing the accuracy and F1-score by only 0.5%. We also show that our approach is both highly transferable across different datasets and adaptable to changes in individual detector performance.
BibTeX
@misc{birman2019transferable,
  author = {Yoni Birman and Shaked Hindi and Gilad Katz and Asaf Shabtai},
  title = {{Transferable Cost-Aware Security Policy Implementation for Malware Detection Using Deep Reinforcement Learning}},
  year = {2019},
  eprint = {1905.10517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.10517},
}
Figure