Toward the Detection of Polyglot Files

Luke Koch, Sean Oesch, Mary Adkisson, Sam Erwin, Brian Weber, Amul ChaulagainarXiv 2022 · DOI · arXiv

TL;DR

This paper tackles the challenge of detecting "polyglot" files – files that masquerade as multiple valid file types simultaneously, often evading standard malware detection. The authors introduce a new dataset of polyglot files and evaluate various machine and deep learning models, finding that a tuned CatBoost model, augmented with file type information, achieves the highest recall for polyglot detection.

What Problem Does This Paper Address?

Modern cybersecurity relies heavily on identifying file types to extract relevant features for malware detection. However, a clever technique called "polyglotting" allows attackers to create files that are valid as, for example, both an image (JPG) and an executable archive (JAR). This dual nature can trick security tools into analyzing only one of the file's identities, allowing malicious code embedded in the other identity to slip through undetected. Existing file identification tools, like the common file utility, struggle with these ambiguous files, leading to a critical gap in malware detection pipelines. This paper aims to bridge that gap by developing a robust method for identifying polyglot files before they are processed by type-specific malware analysis.

Methods

The researchers explored two main approaches:

  • Machine Learning (ML) Models: They trained several traditional ML models (Random Forest, Support Vector Machine, Stochastic Gradient Descent, LightGBM, Gradient Boosting, and CatBoost) using byte histograms as features. A byte histogram is essentially a count of how many times each possible byte value (0-255) appears in a file. This provides a file-type-agnostic representation.
  • Deep Learning (DL) Model: They employed MalConv2, a convolutional neural network (CNN) specifically designed for malware detection. MalConv2 works directly on the raw bytes of a file, allowing it to automatically learn relevant features.
  • Feature Engineering: To improve the ML models, they experimented with concatenating the byte histogram with the output of the file utility (specifically, its MIME type). This leverages the file utility's strength in identifying conventional files while providing ML models with explicit file type hints.

Datasets

The authors created a new dataset specifically for polyglot detection. It comprises:

  • Monoglot (Normal) Files: 7 common file types (PDF, TIFF, PNG, JPG, JAR, ZIP, ISO, GIF, DCM, PE).
  • Polyglot Files: 21 different combinations of the monoglot file types, created using the mitra tool. These polyglots were intentionally created without malicious code to focus solely on the detection challenge and facilitate data sharing.

The dataset was split into an 80/20 train/test ratio.

Key Insights and Ideas

  • The file utility's Achilles' heel: While excellent for standard files, the file utility exhibits very poor recall for polyglot files, especially for "stack" and "parasite" types, highlighting its limitations in this domain.
  • Header-centricity can be a strength: MalConv2's tendency to focus on file headers, often a weakness in malware detection, becomes an advantage for polyglot detection, as headers are key indicators of file type.
  • File type hints matter: Augmenting byte histograms with the MIME type from the file utility significantly boosted the performance of ML models, demonstrating the value of combining traditional file analysis with learned representations.
  • CatBoost shines with added context: The CatBoost model, when enhanced with the MIME type feature, achieved the highest recall, narrowly outperforming the deep learning model MalConv2. This suggests that for this specific task, a well-tuned ML model with carefully selected features can be highly effective.

Key Results

  • MalConv2: Achieved the highest recall (95.16%) and F1 score (95.58%) when trained on raw bytes.
  • CatBoost (with byte histogram): Achieved a recall of 83.86%.
  • CatBoost (with byte histogram + MIME type): Significantly improved recall to 94.75%.
  • Tuned CatBoost (with byte histogram + MIME type): Achieved the best recall of 95.45%.
  • file utility: Demonstrated very poor recall (28.11% overall) on the polyglot test set, though it maintained high precision (99.61%) for monoglot files.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Data Diversification: The current dataset, while comprehensive, needs to be expanded to encompass an even wider variety of file formats and polyglot creation methods.
  • Feature Space Optimization: Further exploration of features like byte co-occurrence counts and byte entropy could potentially improve model performance.
  • Hyper-parameter Tuning: Continued optimization of the top-performing models is necessary to maximize their effectiveness.
  • High Throughput: Demonstrating that these models can operate efficiently enough for real-time, production-grade systems is crucial.
  • Benign Polyglots: Addressing the challenge of distinguishing malicious polyglots from benign ones (e.g., DICOM+TIFF for medical imaging) is an important area for future work.
  • MalConv2's Input Size: The original MalConv2 truncates files larger than 16MB, leading to slower training. Future work could explore optimizations for handling larger files.

Summary updated

Abstract

Standardized file formats play a key role in the development and use of computer software. However, it is possible to abuse standardized file formats by creating a file that is valid in multiple file formats. The resulting polyglot (many languages) file can confound file format identification, allowing elements of the file to evade analysis.This is especially problematic for malware detection systems that rely on file format identification for feature extraction. File format identification processes that depend on file signatures can be easily evaded thanks to flexibility in the format specifications of certain file formats. Although work has been done to identify file formats using more comprehensive methods than file signatures, accurate identification of polyglot files remains an open problem. Since malware detection systems routinely perform file format-specific feature extraction, polyglot files need to be filtered out prior to ingestion by these systems. Otherwise, malicious content could pass through undetected. To address the problem of polyglot detection we assembled a data set using the mitra tool. We then evaluated the performance of the most commonly used file identification tool, file. Finally, we demonstrated the accuracy, precision, recall and F1 score of a range of machine and deep learning models. Malconv2 and Catboost demonstrated the highest recall on our data set with 95.16% and 95.45%, respectively. These models can be incorporated into a malware detector's file processing pipeline to filter out potentially malicious polyglots before file format-dependent feature extraction takes place.
BibTeX
@misc{koch2022toward,
  author = {Luke Koch and Sean Oesch and Mary Adkisson and Sam Erwin and Brian Weber and Amul Chaulagain},
  title = {{Toward the Detection of Polyglot Files}},
  year = {2022},
  eprint = {2203.07561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.07561},
}
Figure