Introduction
The sheer volume of modern malware presents a significant challenge for security researchers and production-scale repositories. With collections often exceeding a billion files and consuming petabytes of storage, traditional analysis methods—such as static analysis of binaries or dynamic execution in sandboxes—face severe scalability bottlenecks. Furthermore, these methods are frequently "format-locked" to specific operating systems (e.g., Windows PE files) and are susceptible to obfuscation techniques.
This paper introduces AVScan2Vec, a novel approach that leverages antivirus (AV) scan reports as a primary data source. By treating the collective labels from dozens of AV vendors as a "language," AVScan2Vec generates compact, semantically rich feature vectors. This method is format-agnostic, highly efficient, and capable of performing similarity searches at production scale with significantly lower memory requirements than existing state-of-the-art features.
Methodology
AVScan2Vec utilizes a Transformer-based architecture designed to parse and embed the noisy, inconsistent strings found in AV scan reports (e.g., VirusTotal outputs).
Preprocessing and Tokenization
AV labels are notoriously heterogeneous. To handle this, the authors decompose labels into tokens (e.g., Ransom.Win32.Wanna.xyz!gen becomes ransom, win32, wanna). To manage the vast vocabulary of unique strings without a fixed dictionary, the model employs a CharacterBERT approach, which learns representations from character sequences, providing robustness against minor variations and typos.
Training Objectives
The model undergoes a two-stage training process:
- Self-Supervised Pre-training: Using 37 million reports, the model performs two tasks:
- Masked Token Prediction: Predicting hidden tokens within a single vendor's label.
- Masked Label Prediction: Reconstructing an entire missing label from one vendor based on the context provided by other vendors in the same report.
- Fine-tuning: To optimize the embeddings for similarity tasks, the authors use a Siamese network configuration with Multiple Negatives Ranking (MNR) loss. This encourages the model to cluster vectors of malware with the same content hash while separating unrelated samples.
Scalable Search
For deployment, the embeddings are integrated with Dynamic Continuous Indexing (DCI). This allows for sub-linear time nearest-neighbor lookups, enabling analysts to query massive datasets in milliseconds.
Experimental Setup
The authors evaluated AVScan2Vec using several large-scale datasets:
- Pre-training: 37.1 million scan reports from VirusTotal and VirusShare, spanning 2006 to 2023.
- Family Classification: The MOTIF dataset (~3,000 samples) was used to benchmark the model's ability to identify specific malware families.
- Scale and Performance: A 7.3-million-file subset of SOREL-20M was used to test behavioral classification and search speed.
Results
AVScan2Vec demonstrates significant improvements over traditional feature sets like EMBER and BWMD.
Classification and Clustering
On the MOTIF benchmark, AVScan2Vec achieved a family classification accuracy of 80.49%, outperforming the next best method (EMBER-N) by over 6 percentage points. The model also showed superior performance in clustering tasks, effectively grouping related malware families even when vendor labels were inconsistent.
Efficiency and Speed
One of the most striking results is the model's compactness. The feature vectors for 7.3 million files required only 22.4 GB of storage—a 3x reduction compared to EMBER (69.4 GB) and an 85x reduction compared to BWMD (1.9 TB). This efficiency allows the entire index to reside in RAM. Using DCI, the system achieved a nearest-neighbor search time of 16.41 milliseconds across the 7.19 million sample index.
Robustness
The model proved resilient to "future" malware. When tested on samples released after the training data cutoff, AVScan2Vec maintained high performance, suggesting it learns the underlying semantics of malicious behavior rather than simply memorizing specific strings.
Discussion and Limitations
AVScan2Vec's primary advantage is its format-agnostic nature; because it relies on scan reports, it can process Windows, Android, and Linux malware with equal efficacy. However, the authors note that the malware landscape evolves rapidly, likely requiring monthly retraining to mitigate concept drift. Future work will focus on scaling the Transformer architecture further and evaluating the model's performance on mixed datasets containing both benign and malicious files.
Conclusion
AVScan2Vec represents a significant step forward in production-scale malware analysis. By transforming noisy AV scan data into high-quality embeddings, it enables rapid, accurate, and memory-efficient classification and search, providing a powerful tool for analysts managing billion-file repositories.
Summary updated