TL;DR
EMBERSim bridges a critical gap in cybersecurity research by providing a massive, metadata-rich dataset for binary code similarity. By augmenting the well-known EMBER dataset with fine-grained malware tags and introducing a novel "leaf similarity" search method based on XGBoost, the authors provide a scalable way to hunt for related files across both malicious and benign samples.
The Challenge of Binary Code Similarity
In the cat-and-mouse game of malware analysis, attackers often create "new" threats by slightly modifying existing code to bypass signature-based detection. To counter this, researchers use Binary Code Similarity (BCS)—the art of finding "near-duplicates" without access to the original source code.
However, the field faces two major hurdles:
- Data Scarcity: Most BCS datasets are small or focus exclusively on malicious files, ignoring the "clean" (benign) data that is essential for avoiding false positives.
- Granularity: Existing tools often look at individual functions rather than whole programs, making it hard to get a "big picture" view of a file's behavior.
To address these issues, this paper introduces EMBERSim, a dataset of 1 million Windows Portable Executable (PE) files, providing the community with a robust benchmark for whole-program similarity search.
Methods: Leaf Similarity and Tag Enrichment
The authors contribute both a comprehensive dataset and a specific methodology for similarity search.
- Leaf Similarity: Instead of using traditional "fuzzy hashing" (like
ssdeep), the authors repurpose an XGBoost malware classifier. When a file passes through the ensemble of trees in the model, it ends up in a specific "leaf" for each tree. The similarity between two files is calculated by counting how many times they land in the same leaf nodes. If two files follow the same paths through the forest, they are likely structurally and behaviorally similar. - Tag Enrichment & Ranking: To create a "ground truth" for what makes files similar, the authors used AVClass v2 to extract family (FAM), class (CLASS), and behavior (BEH) tags from VirusTotal reports. They developed a custom ranking algorithm (using tag co-occurrence) to filter out noisy or conflicting labels from different antivirus vendors, ensuring the metadata is as accurate as possible.
Datasets
- EMBER (2018): The foundation of this work. A dataset of 1 million PE file features (800k labeled, 200k unlabeled).
- EMBERSim (New): The authors' augmented version of EMBER. It includes the original features plus standardized malware tags and "similarity-informed" labels.
- DREBIN: Used as an external dataset to prove that their "leaf similarity" method generalizes well to other types of malware data.
Key Insights
- Classifiers as Feature Extractors: The most clever insight is realizing that a well-trained malware classifier has already learned the "essence" of what makes a file malicious. By using the leaf indices (proximities) as a hash, they leverage the model's internal logic to perform similarity searches without needing a separate embedding model.
- The "Clean Data" Blind Spot: Most similarity research ignores benign files. The authors argue that quantifying similarity in clean data is vital for understanding "detection bypass"—where a malicious file tries to look like a legitimate one.
- Tag Co-occurrence: Since antivirus vendors often disagree on names (e.g., one calls a file "WannaCry" and another calls it "Ransomware.A"), the authors used statistical co-occurrence to map these disparate names to a unified, ranked set of tags.
Key Results
The proposed leaf similarity method demonstrated strong performance across several evaluations:
- Superiority over ssdeep: In "Label Homogeneity" tests (measuring how often the top-K results share the same label as the query), the leaf similarity method achieved near-perfect scores ($1.0$ for Top-1), significantly outperforming the industry-standard
ssdeeptool, which struggled particularly with benign files. - High Relevance: In a "Counterfactual Analysis" (simulating a real-world production environment), the method retrieved benign similar samples with over 95% accuracy and malicious samples with over 71% relevance even for the top 100 hits.
- Generalization: When tested on the DREBIN dataset, the leaf similarity method maintained high performance, proving it isn't just "overfitting" to the EMBER data.
Limitations and Future Work
- The "Benign" Assumption: The authors acknowledge a potential bias: they treat the absence of tags as an indicator that a sample is benign. In the real world, a new, undetected malware sample might also have no tags.
- Vendor Noise: The ground truth relies on VirusTotal detections. If the majority of AV vendors are wrong about a sample, the EMBERSim tags will reflect that error.
- Model Dependency: The similarity method is tied to the specific XGBoost model. If the model is updated or retrained, the "leaf hashes" change, meaning the indexed database would need to be refreshed.
- Future Directions: The authors suggest exploring multiple tree-based models and improving "model interpretability"—essentially trying to explain why the forest thinks two files are similar.
Summary updated