Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing

Daniel Gibert, Giulio Zizzo, Quan Le

Introduction

Deep learning (DL) has revolutionized malware detection, providing automated feature extraction and high accuracy against known threats. However, these models possess a significant vulnerability: they are highly susceptible to adversarial evasion attacks. In the context of malware, an attacker can make minor, non-functional modifications to a malicious executable—such as appending benign code or filling unused "code caves" with junk data—to trick a DL-based detector into classifying it as benign.

In the paper "Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing," researchers Daniel Gibert, Giulio Zizzo, and Quan Le propose a novel defense mechanism called Randomized Ablation-based Smoothing (RA). This approach adapts the concept of randomized smoothing from computer vision to the discrete, categorical domain of binary files to create more resilient malware detectors.

The Problem: Evasion in the Binary Domain

Static malware detectors, such as the widely used MalConv architecture, analyze the raw byte stream of an executable. While effective, these models often over-fit to specific byte sequences. Attackers exploit this by injecting "benign-looking" noise that does not alter the file's malicious functionality but shifts the model's internal representation toward the benign class.

Traditional defenses like Gaussian smoothing, which add continuous noise to input features, are inapplicable here because malware binaries are composed of discrete bytes (0-255). Changing a byte's value by a small numerical increment (e.g., changing a 0x01 to a 0x02) can fundamentally change the instruction or data, potentially breaking the file or creating nonsense inputs for the model.

Methodology: Randomized Ablation-based Smoothing

To address the categorical nature of binaries, the authors propose Randomized Ablation. Instead of adding noise, the system "blurs" the input by randomly deleting portions of it.

1. Training Phase

During training, the base classifier (e.g., MalConv) is not trained on the original files. Instead, it is trained on ablated versions where a fixed percentage ($p$) of bytes are randomly replaced with a special 'PAD' token. This forces the model to learn the core indicators of maliciousness that persist even when large portions of the file are missing, preventing it from relying on fragile, easily manipulated byte sequences.

2. Inference (Testing) Phase

When a new file is presented for classification, the system does not rely on a single pass. Instead:

  • It generates $L$ different ablated versions of the input file.
  • The base classifier predicts a label for each of the $L$ versions.
  • The final classification is determined by a majority vote across all $L$ predictions.

This statistical approach ensures that even if an attacker successfully injects an adversarial payload, there is a high probability that the payload will be "ablated" away in a significant number of the $L$ versions, allowing the underlying malware signature to be detected.

Experimental Results

The researchers evaluated the RA-MalConv model using the BODMAS dataset, which contains over 134,000 Windows PE files. The smoothed model was tested against several state-of-the-art evasion attacks, with the following key findings:

  • Resilience to Code Cave Attacks: This attack hides malicious intent in unused file spaces. While a standard non-smoothed (NS) model saw its detection rate plummet to a mere 14%, the RA-smoothed model maintained a robust 81% detection rate.
  • Preservation of Clean Accuracy: A common drawback of security hardening is a decrease in performance on legitimate data. However, the RA model achieved approximately 97.6% accuracy on clean files, nearly identical to the baseline model.
  • Model Agnosticism: Because RA acts as a wrapper around the input and output, it can be applied to virtually any machine learning-based malware detector without requiring changes to the underlying architecture.

Trade-offs and Limitations

The primary cost of this increased security is computational overhead. Because the model must perform $L$ passes (the study used values like 20 or 100) to reach a majority verdict, inference time is significantly higher. In testing, the smoothed model took approximately 3.38 seconds per file, compared to 0.016 seconds for the baseline. While this may be acceptable for deep-scan email gateways or forensic analysis, it may pose challenges for real-time endpoint protection without hardware acceleration.

Conclusion

Randomized Ablation-based Smoothing represents a significant step toward practical, mathematically grounded defenses for malware detection. By treating the binary as a collection of discrete entities and utilizing a majority-vote mechanism, the RA framework effectively mitigates the impact of adversarial injections. This research demonstrates that robustness in deep learning does not necessarily require sacrificing accuracy, provided the defense is tailored to the specific constraints of the data domain.

Summary updated

Abstract

Malware detectors based on deep learning (DL) have been shown to be susceptible to malware examples that have been deliberately manipulated in order to evade detection, a.k.a. adversarial malware examples. More specifically, it has been show that deep learning detectors are vulnerable to small changes on the input file. Given this vulnerability of deep learning detectors, we propose a practical defense against adversarial malware examples inspired by randomized smoothing. In our work, instead of employing Gaussian or Laplace noise when randomizing inputs, we propose a randomized ablation-based smoothing scheme that ablates a percentage of the bytes within an executable. During training, our randomized ablation-based smoothing scheme trains a base classifier based on ablated versions of the executable files. At test time, the final classification for a given input executable is taken as the class most commonly predicted by the classifier on a set of ablated versions of the original executable. To demonstrate the suitability of our approach we have empirically evaluated the proposed ablation-based model against various state-of-the-art evasion attacks on the BODMAS dataset. Results show greater robustness and generalization capabilities to adversarial malware examples in comparison to a non-smoothed classifier.
BibTeX
@inproceedings{gibert2023towards,
  author = {Daniel Gibert and Giulio Zizzo and Quan Le},
  title = {{Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing}},
  booktitle = {ESORICS Workshops},
  year = {2023},
  url = {https://openreview.net/forum?id=TMNOgunpt1},
}
Figure