Abstract
Dynamic malware analysis executes programs in isolated environments to monitor run-time behavior, such as system API calls, for detection. While effective against obfuscation and zero-day threats, existing methods often ignore API arguments or rely on complex, expert-heavy feature engineering. This paper proposes a low-cost feature extraction approach using a feature hashing trick to encode API call arguments. We introduce a deep neural network architecture utilizing Gated-CNNs to identify local patterns and bidirectional LSTMs to capture sequential correlations. Experimental results on a large real-world dataset demonstrate that our solution significantly outperforms baselines, achieving high accuracy with minimal computational overhead.
Introduction
Malware detection is a constant arms race. Static analysis—scanning code without execution—is frequently bypassed by sophisticated obfuscation and zero-day exploits. Dynamic analysis offers a more robust alternative by observing a program's actual behavior in a secure sandbox. However, a significant gap remains: most tools focus solely on API names (e.g., RegSetValue) while ignoring the rich context provided by API arguments (e.g., the specific registry key being modified). Processing these arguments is challenging because they are heterogeneous, ranging from integers and file paths to URLs and IP addresses.
System Architecture
The proposed framework follows a comprehensive pipeline from initial file collection to final classification. As shown in the system overview, the process begins with PE (Portable Executable) file collection, followed by behavioral monitoring in a virtual machine cluster to generate API logs. These logs are then processed through our feature extraction and deep learning training modules.
Methodology
Feature Hashing for Heterogeneous Data
To handle the complexity of API arguments without manual feature engineering, we employ a "feature hashing trick." This approach transforms diverse data types into a compact, fixed-length 102-dimension feature vector for every API call:
- Integers: Normalized using a log scale to mitigate the impact of outliers.
- Strings: Processed hierarchically. For instance, a file path like
C:\a\b\cis hashed into multiple components (C:,C:\a, etc.) to preserve directory structure context.
Deep Learning Architecture
The core of our detection engine is a hybrid neural network designed to capture both local and global behavioral patterns.
- Gated-CNNs: Multiple Gated Convolutional Neural Networks with varying kernel sizes (2 and 3) act as n-gram detectors, identifying short-term patterns in the API sequence.
- Bi-LSTM: The CNN outputs are fed into a Bidirectional Long Short-Term Memory network. This layer is crucial for capturing long-term dependencies, allowing the model to understand the "story" of a program's execution by linking early actions to later behaviors.
- Global Max-Pooling: This layer extracts the most significant signals across the entire execution sequence before passing them to the final dense layers for classification.
Experimental Evaluation
Dataset and Environment
The model was evaluated using a massive real-world dataset from SecureAge Technology. Training and cross-validation were performed on 27,348 files from April, while 33,839 files from May were used for testing to simulate the detection of new, unseen malware. All files were executed in a Cuckoo Sandbox environment running 64-bit Windows 7.
Key Results
Our model demonstrated superior performance across all critical metrics compared to established baselines:
- AUC (Area Under Curve): 98.71% on the test set.
- Accuracy: 95.33%.
- Recall at 0.1% FPR: 71.48%. This low False Positive Rate is vital for industrial applications where "false alarms" can disrupt legitimate user activity.
- Efficiency: The total inference time is approximately 129ms per sample, making it highly suitable for real-time sandbox environments.
Conclusion
By effectively incorporating API arguments through feature hashing and leveraging the combined strengths of Gated-CNNs and Bi-LSTMs, we have developed a high-performance framework for dynamic malware detection. Our findings confirm that API arguments are a "smoking gun" for identifying malicious intent. Future work will explore more complex data types, such as C-style structs, and investigate defenses against adversarial "anti-sandbox" techniques.
Summary updated