TL;DR
This paper tackles the challenge of detecting "polyglot" files – files that masquerade as multiple valid file types simultaneously, often evading standard malware detection. The authors introduce a new dataset of polyglot files and evaluate various machine and deep learning models, finding that a tuned CatBoost model, augmented with file type information, achieves the highest recall for polyglot detection.
What Problem Does This Paper Address?
Modern cybersecurity relies heavily on identifying file types to extract relevant features for malware detection. However, a clever technique called "polyglotting" allows attackers to create files that are valid as, for example, both an image (JPG) and an executable archive (JAR). This dual nature can trick security tools into analyzing only one of the file's identities, allowing malicious code embedded in the other identity to slip through undetected. Existing file identification tools, like the common file utility, struggle with these ambiguous files, leading to a critical gap in malware detection pipelines. This paper aims to bridge that gap by developing a robust method for identifying polyglot files before they are processed by type-specific malware analysis.
Methods
The researchers explored two main approaches:
- Machine Learning (ML) Models: They trained several traditional ML models (Random Forest, Support Vector Machine, Stochastic Gradient Descent, LightGBM, Gradient Boosting, and CatBoost) using byte histograms as features. A byte histogram is essentially a count of how many times each possible byte value (0-255) appears in a file. This provides a file-type-agnostic representation.
- Deep Learning (DL) Model: They employed MalConv2, a convolutional neural network (CNN) specifically designed for malware detection. MalConv2 works directly on the raw bytes of a file, allowing it to automatically learn relevant features.
- Feature Engineering: To improve the ML models, they experimented with concatenating the byte histogram with the output of the
fileutility (specifically, its MIME type). This leverages thefileutility's strength in identifying conventional files while providing ML models with explicit file type hints.
Datasets
The authors created a new dataset specifically for polyglot detection. It comprises:
- Monoglot (Normal) Files: 7 common file types (PDF, TIFF, PNG, JPG, JAR, ZIP, ISO, GIF, DCM, PE).
- Polyglot Files: 21 different combinations of the monoglot file types, created using the
mitratool. These polyglots were intentionally created without malicious code to focus solely on the detection challenge and facilitate data sharing.
The dataset was split into an 80/20 train/test ratio.
Key Insights and Ideas
- The
fileutility's Achilles' heel: While excellent for standard files, thefileutility exhibits very poor recall for polyglot files, especially for "stack" and "parasite" types, highlighting its limitations in this domain. - Header-centricity can be a strength: MalConv2's tendency to focus on file headers, often a weakness in malware detection, becomes an advantage for polyglot detection, as headers are key indicators of file type.
- File type hints matter: Augmenting byte histograms with the MIME type from the
fileutility significantly boosted the performance of ML models, demonstrating the value of combining traditional file analysis with learned representations. - CatBoost shines with added context: The CatBoost model, when enhanced with the MIME type feature, achieved the highest recall, narrowly outperforming the deep learning model MalConv2. This suggests that for this specific task, a well-tuned ML model with carefully selected features can be highly effective.
Key Results
- MalConv2: Achieved the highest recall (95.16%) and F1 score (95.58%) when trained on raw bytes.
- CatBoost (with byte histogram): Achieved a recall of 83.86%.
- CatBoost (with byte histogram + MIME type): Significantly improved recall to 94.75%.
- Tuned CatBoost (with byte histogram + MIME type): Achieved the best recall of 95.45%.
fileutility: Demonstrated very poor recall (28.11% overall) on the polyglot test set, though it maintained high precision (99.61%) for monoglot files.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Data Diversification: The current dataset, while comprehensive, needs to be expanded to encompass an even wider variety of file formats and polyglot creation methods.
- Feature Space Optimization: Further exploration of features like byte co-occurrence counts and byte entropy could potentially improve model performance.
- Hyper-parameter Tuning: Continued optimization of the top-performing models is necessary to maximize their effectiveness.
- High Throughput: Demonstrating that these models can operate efficiently enough for real-time, production-grade systems is crucial.
- Benign Polyglots: Addressing the challenge of distinguishing malicious polyglots from benign ones (e.g., DICOM+TIFF for medical imaging) is an important area for future work.
- MalConv2's Input Size: The original MalConv2 truncates files larger than 16MB, leading to slower training. Future work could explore optimizations for handling larger files.
Summary updated