To believe or not to believe: Validating explanation fidelity for dynamic malware analysis

Li Chen, Carter Yagemann, Evan DowningWorkshops @ CVPR 2019

TL;DR

This paper explores how well "explainable AI" techniques can validate the predictions of deep learning models used for dynamic malware analysis. By applying an image-based explanation method to malware visualizations, the authors demonstrate that these explanations can indeed highlight suspicious code patterns, offering valuable insights for security analysts.

What Problem Does This Paper Address?

Malware detection is a critical cybersecurity challenge. While deep learning models, particularly those using computer vision on malware-generated images, have shown impressive accuracy, understanding why a model flags something as malicious is crucial for trust and effective response. Traditional static analysis can be thwarted by code obfuscation, making dynamic analysis (observing malware in action) more robust. However, interpreting the complex behavior captured by dynamic analysis and visualized as images is difficult. This paper tackles the problem of validating the "fidelity" of explanations generated by AI models for these dynamic malware images, ensuring the explanations align with actual security expertise.

Methods

The core of this paper involves extending the Local Interpretable Model-agnostic Explanations (LIME) framework, a popular technique for explaining AI predictions on natural images, to the domain of dynamic malware classification. LIME works by creating local approximations of the model's decision boundary around a specific prediction, highlighting the input features (in this case, pixels in the malware image) that most influence the outcome.

The authors employ deep transfer learning using pre-trained VGG models on ImageNet. They then retrain these models on malware images generated from dynamic analysis traces. For explanation, they apply LIME to these trained models and then critically assess the generated "heatmaps" or highlighted regions. The validation of these explanations relies heavily on security domain expertise, where researchers manually inspect the code corresponding to the highlighted image regions to see if they reveal known malicious behaviors or exploit patterns.

Datasets

The paper does not explicitly list a single, unified dataset. Instead, it describes the data generation and collection processes for its two case studies:

  • Case Study I (pdfka family): Traces were collected from Adobe Acrobat Reader opening PDF documents. Benign and malicious traces were generated, then disassembled into basic blocks (BBIDs). These BBIDs were then used to generate images based on prediction correctness from an LSTM model.
  • Case Study II: A dataset of 13,394 Windows Portable Executable (PE) samples (malicious) was collected from the Georgia Tech Research Institute (GTRI) using their APIARY platform. A benign dataset of 5,772 samples was collected by crawling CNET. These samples were executed for 2 minutes in a controlled environment (Windows 7 32-bit KVM) to capture dynamic features. Three types of images were generated from these dynamic features: API existence, API sequence, and API frequency.

Key Insights and Ideas

  • Bridging Vision and Security Expertise: The paper highlights the critical need to validate AI explanations not just visually, but through the lens of domain experts. This is a significant step beyond applying general-purpose explanation techniques to specialized security problems.
  • Indirect Calls as Explanations: In the first case study, LIME successfully identified indirect calls that were unique to a specific malware family's exploit behavior, demonstrating the potential for explanations to pinpoint exploitable functionality.
  • API Feature Sensitivity: The effectiveness of image-based explanations varied significantly depending on how the dynamic behavior was represented. Explanations for API existence were insightful (e.g., highlighting crypto-related APIs), while explanations for API sequences and frequencies were often ambiguous. This suggests that the choice of feature representation for image generation is crucial for effective explanation.
  • Interpretation as a Validation Dimension: The authors strongly advocate for interpretation fidelity as a key metric for evaluating malware classifiers, alongside traditional accuracy metrics.

Key Results

  • Case Study I: The interpretation framework successfully identified indirect calls that were manually confirmed to be characteristic of the pdfka malware family's exploit behavior (specifically related to CVE-2010-0188). The model achieved 100% classification accuracy on the test set.
  • Case Study II: Deep transfer learning models achieved approximately 95% accuracy across three different image generation methods (API existence, sequence, frequency).
  • Explanations for API existence and frequency images provided insights into common malware communication patterns (e.g., C&C servers).
  • Explanations for API sequence images were found to be ambiguous, even when the classifier was confident, suggesting API sequence alone might be a weaker feature for interpretable image generation.

Limitations and Future Work

  • Ambiguity in API Sequence Explanations: The paper acknowledges that explanations for API sequence-based images were ambiguous, indicating a need for further research into how to generate more meaningful interpretations from such data.
  • Benign Dataset Limitations: The authors note that their benign dataset might be missing certain types of software, which could lead to misinterpretations or over-reliance on common internet browser behaviors.
  • Future Work: The authors plan to continue developing and proposing interpretation schemes specifically tailored for image-based malware classification frameworks, aiming to improve their robustness and applicability in real-world security scenarios. They also emphasize the ongoing need to combine machine learning with domain expertise for effective validation.

Summary updated

BibTeX
@inproceedings{chen2019believe,
  author = {Li Chen and Carter Yagemann and Evan Downing},
  title = {{To believe or not to believe: Validating explanation fidelity for dynamic malware analysis}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure