Dynamic Malware Analysis with Feature Engineering and Feature Learning

Zhaoqi Zhang, Panpan Qi, Wei Wang

Introduction

Malware detection is a critical pillar of cybersecurity, traditionally divided into static and dynamic analysis. While static analysis examines code without execution, dynamic analysis monitors a program's behavior in a controlled environment. Dynamic analysis is inherently more resilient to code obfuscation and "zero-day" threats, but it generates massive amounts of complex, heterogeneous data that is difficult for standard machine learning models to process efficiently.

This paper introduces a high-performance framework for dynamic malware detection that focuses on the often-overlooked importance of API arguments. By combining a "hashing trick" to process messy API data with a hybrid Gated-CNN and Bi-LSTM neural network, the researchers achieved a 98.71% AUC, significantly outperforming models that rely solely on API names.

The Challenge of API Arguments

Every time a program makes a system call (API call), it generates a name (e.g., RegOpenKey) and a set of arguments (e.g., file paths, registry keys, integers, or IP addresses). Most existing deep learning models ignore these arguments because they are "heterogeneous"—a mix of different data types that are hard to feed into a standard neural network. Previous attempts to include them often required labor-intensive manual feature engineering or treated everything as a simple string, losing vital context. However, arguments are highly discriminative: writing to a temporary log is benign, but writing to a system DLL is a clear indicator of malicious intent.

System Architecture

The proposed framework follows a structured pipeline from raw file collection to final classification. The process begins with collecting Portable Executable (PE) files, which are then executed in a Cuckoo Sandbox environment to generate detailed API logs. These logs are then processed through a feature engineering stage before being fed into the deep learning model.

The overall pipeline of the proposed dynamic malware analysis framework, from file collection to model training.

Methodology

The authors propose a two-part solution: a low-cost feature engineering step followed by a specialized deep learning architecture.

1. Feature Engineering: The Hashing Trick

To handle the variety of API arguments without manual rules for every possible case, the authors used feature hashing. This technique maps features to a fixed-size vector using a hash function, allowing the model to remain "low-cost" and scalable.

  • API Names and Categories: Split into words/characters and hashed.
  • Integers: Normalized using a logarithm to handle sparse distributions.
  • Strings (Paths, URLs, IPs): Parsed into hierarchical substrings (e.g., C:\a\b becomes C:, C:\a, and C:\a\b) and then hashed.
  • Statistical Features: The model includes 10 global statistics, such as string entropy and the count of "MZ" headers.

The result is a compact 102-dimension feature vector for every single API call in the execution trace.

2. Hybrid Model Architecture

The core of the framework is a hybrid neural network designed to capture both local patterns and long-term temporal dependencies in API sequences.

Detailed architecture of the hybrid Gated-CNN and Bi-LSTM model used for classification.

As shown in the architecture diagram, the model processes the sequence of 102-d vectors through three main stages:

  • Gated-CNN Module: Two parallel Gated Convolutional Neural Networks (with kernel sizes of 2 and 3) act as "n-gram detectors." The "gate" mechanism allows the model to selectively pass only the most relevant behavioral information, filtering out noise more efficiently than standard CNNs.
  • Bi-LSTM Module: A Bidirectional Long Short-Term Memory network captures the sequential relationship between calls, looking at both past and future context in the execution trace. This is vital because the order of operations is as important as the operations themselves.
  • Classification Module: Global max-pooling and dense layers produce the final probability of the file being malicious.

Experimental Results

The researchers utilized a massive, real-world dataset provided by SecureAge Technology, consisting of approximately 27,000 PE files for training and 33,000 files for testing. Testing on a later month's data ensured the model could handle evolving malware trends.

The proposed model outperformed all baselines, including traditional machine learning (SVM, Random Forest) and other deep learning approaches (standard RNNs and CNNs):

  • AUC: 98.71% on the test set.
  • Accuracy: 95.33%.
  • Recall at 0.1% False Positive Rate: 71.48%. This is a critical metric for security products, as low false-alarm rates are essential to prevent "alert fatigue."
  • Inference Speed: The model processes samples in roughly 129ms, making it suitable for near-real-time deployment.

Conclusion and Future Work

By leveraging the discriminative power of API arguments through feature hashing and a hybrid deep learning architecture, this framework sets a new standard for dynamic analysis. While highly effective, the authors note that some complex C-style structs are still not fully captured. Future work will explore these ignored argument types and investigate the model's robustness against adversarial attacks designed to confuse sequential models with "junk" API calls.

Reference: Zhang, Z., Qi, P., & Wang, W. (2020). Dynamic Malware Analysis with Feature Engineering and Feature Learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-20).

Summary updated

Abstract

Dynamic malware analysis executes the program in an isolated environment and monitors its run-time behaviour (e.g. system API calls) for malware detection. This technique has been proven to be effective against various code obfuscation techniques and newly released ("zero-day") malware. However, existing works typically only consider the API name while ignoring the arguments, or require complex feature engineering operations and expert knowledge to process the arguments. In this paper, we propose a novel and low-cost feature extraction approach, and an effective deep neural network architecture for accurate and fast malware detection. Specifically, the feature representation approach utilizes a feature hashing trick to encode the API call arguments associated with the API name. The deep neural network architecture applies multiple Gated-CNNs (convolutional neural networks) to transform the extracted features of each API call. The outputs are further processed through bidirectional LSTM (long-short term memory networks) to learn the sequential correlation among API calls. Experiments show that our solution outperforms baselines significantly on a large real dataset. Valuable insights about feature engineering and architecture design are derived from the ablation study.
BibTeX
@inproceedings{zhang2020dynamic,
  author = {Zhaoqi Zhang and Panpan Qi and Wei Wang},
  title = {{Dynamic Malware Analysis with Feature Engineering and Feature Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5474},
}
Figure