Abstract
Assembly code search is vital for reducing the burden on reverse engineers, allowing them to quickly identify specific functions using natural language within vast binary programs. Despite its significance, this critical task is impeded by the complexities involved in building high-quality datasets. This paper explores training a Large Language Model (LLM) to emulate a general compiler. By leveraging Ubuntu packages to compile a dataset of 20 billion tokens, we further continue pre-train CodeLlama as a Virtual Compiler (ViC), capable of compiling any source code of any language to assembly code. This approach allows for virtual compilation across a wide range of programming languages without the need for a real compiler, preserving semantic equivalency and expanding the possibilities for assembly code dataset construction. Furthermore, we use ViC to construct a sufficiently large dataset for assembly code search. Employing this extensive dataset, we achieve a substantial improvement in assembly code search performance, with our model surpassing the leading baseline by 26%.
Introduction
Reverse engineers often need to find specific functions, such as encryption routines or malicious payloads, within massive binary files. While high-level code search (searching Python or Java with natural language) is a relatively mature field, assembly code search presents unique challenges.
The primary bottleneck is data. Training a model to understand the relationship between natural language and assembly requires a dataset consisting of [Natural Language ↔ Source Code ↔ Assembly]. While [Natural Language ↔ Source Code] pairs are abundant due to docstrings in public repositories, the step to generate assembly—compilation—is a significant hurdle. Real-world compilers require specific headers, libraries, and dependencies that are often missing for isolated code snippets. Previous attempts to compile millions of programs often saw success rates as low as 1%, leading to insufficient datasets.
Methodology: The Virtual Compiler (ViC)
To address the "compilation wall," this research introduces ViC (Virtual Compiler), an LLM fine-tuned to emulate a general compiler. Instead of relying on brittle physical environments, ViC allows researchers to "virtually compile" source code into assembly.
The process involves three key stages:
- Virtual Compilation Training: CodeLlama-34B was fine-tuned on 15 million source-to-assembly pairs extracted from Ubuntu packages. The model learns to output assembly based on source code and specific flags (like optimization levels O0-O3).
- Data Augmentation: ViC is used to generate assembly for existing large-scale source code datasets (e.g., CodeSearchNet). This allows for the creation of assembly data for languages like Go and Python, even though the model was primarily trained on C/C++.
- Contrastive Learning: Using this "Virtual Assembly," a dual-encoder model is trained. One encoder processes natural language queries, and the other (a Roformer-base model) processes assembly, using InfoNCE loss to align matching pairs in vector space.
Key Insights
- LLM as an Emulator: An LLM can learn the underlying logic of compilation. By training on different optimization levels, ViC learns how code structure changes while preserving semantic meaning.
- Cross-Language Generalization: Surprisingly, ViC can effectively "compile" languages like Go and Python. Since the base CodeLlama model already understands these languages, the fine-tuning simply teaches it the "output format" of x86 assembly.
- Semantic Over Literal Accuracy: While ViC may not produce a byte-for-byte match of a real compiler (e.g., exact stack offsets), the semantic flow is accurate enough to be highly effective for search tasks.
Experimental Results
The model trained on ViC-augmented data, known as HybridAssembly, was evaluated against several state-of-the-art embedding models.
As shown in the results, HybridAssembly significantly outperformed leading baselines, including OpenAI’s text-embedding-3-large and Voyage AI’s specialized code models, by 26% in Recall@1. Additionally, the virtual assembly achieved a BINSIM (Binary Similarity) score of over 0.8, indicating it is nearly indistinguishable from real binary code for downstream analysis tasks.
Conclusion
ViC demonstrates that an LLM-based virtual compiler can overcome the data scarcity issues that have long hindered assembly code search. By preserving semantic equivalency across languages and optimization levels, ViC enables the construction of high-quality, large-scale datasets that dramatically improve the ability of reverse engineers to search binary code using natural language.
Summary updated