Learning Cross-Architecture Instruction Embeddings for Binary Code Analysis in Low-Resource Architectures

Junzhe Wang, Qiang Zeng, Lannan LuoNAACL 2024 · DOI

TL;DR

This paper introduces CrossIns2Vec, a novel method for learning instruction embeddings that are consistent across different Instruction Set Architectures (ISAs). This allows models trained on data-rich ISAs (like x86) to be effectively transferred to low-resource ISAs (like PowerPC), addressing the critical "data scarcity" problem in binary code analysis.

What Problem Does This Paper Address?

Binary code analysis is crucial for various software security tasks, from malware detection to vulnerability discovery. However, modern software is often compiled for multiple ISAs (e.g., x86, ARM, MIPS, PowerPC). This cross-compilation means vulnerabilities can spread across devices using different ISAs. Analyzing binaries across these ISAs is challenging due to their inherent differences in instruction sets, register usage, and more.

A major hurdle is the "data scarcity" problem: while some ISAs (like x86) have abundant labeled datasets (e.g., for malware), others (like PowerPC) have very few or none. This leads to a disproportionate focus on high-resource ISAs, neglecting low-resource ones. Manually collecting and labeling data for these low-resource ISAs is labor-intensive and time-consuming. Existing deep learning approaches for binary analysis typically require task-specific data for each ISA, exacerbating this problem.

Methods

The core idea is to learn "cross-architecture instruction embeddings" (CAIE) where semantically similar instructions, regardless of their ISA, are mapped to close points in a shared vector space. This shared space acts as a bridge, enabling knowledge transfer between ISAs.

The authors propose a supervised model called CrossIns2Vec. They treat instructions as "words" and basic blocks as "sentences," drawing inspiration from cross-lingual word embeddings in Natural Language Processing (NLP).

Key methodological aspects include:

  • Semantically-Equivalent Basic Block Pairs: The ground truth for semantic equivalence is derived from source code line numbers. Basic blocks from different ISAs are considered equivalent if they originate from the same start and end source code lines. This is facilitated by using compiler flags that embed debug information (like source line numbers) into binaries.
  • Supervised Learning: Unlike some prior unsupervised approaches, CrossIns2Vec uses a supervised model. This is justified by the observation that cross-architecture signals are readily available in binary code analysis (due to cross-compilation), unlike the often difficult-to-obtain cross-lingual signals in NLP.
  • Joint Training Objectives: The model is trained with two objectives:
  • Mono-Architecture Objective: Similar instructions within the same ISA should have close embeddings. This is achieved by predicting a masked instruction using the context of other instructions in the same basic block.
  • Cross-Architecture Objective: Similar instructions across different ISAs should have close embeddings. This is done by predicting a masked instruction in one ISA's basic block using the context of instructions from a semantically equivalent basic block in another ISA.
  • Instruction Normalization: To handle variations, instructions are normalized by replacing number constants with 0, string literals with <STR>, function names with <FOO>, and other symbols with <TAG>.

Datasets

The paper utilizes several datasets:

  • For Learning CAIE: Datasets are constructed by compiling open-source programs (OpenSSL, Binutils, Curl, etc.) for four ISAs: x86, ARM, MIPS, and PowerPC (PPC). x86 is treated as the high-resource ISA, and ARM, MIPS, and PPC as low-resource.
  • Dx86↔ARM: 2,058,484 semantically-equivalent basic block pairs.
  • Dx86↔MIPS: 2,121,125 semantically-equivalent basic block pairs.
  • Dx86↔PPC: 2,189,139 semantically-equivalent basic block pairs. These datasets are used for training CrossIns2Vec.

  • For Downstream Tasks:

    • Function Similarity Detection: Training dataset on x86 consists of 50,000 similar and 50,000 dissimilar function pairs. Testing datasets for ARM, MIPS, and PPC each contain 5,000 similar and 5,000 dissimilar function pairs. These are derived from different programs than the training set to ensure no overlap.
  • Malware Detection: Malware samples are collected from VirusShare.com. The x86 malware samples are used for training. Testing datasets are created for ARM, MIPS, and PPC. The paper notes significant effort in collecting malware samples for MIPS and PPC.

Key Insights and Ideas

  • Bridging the ISA Gap: The core insight is that semantically similar instructions can be mapped to a common embedding space, effectively creating a "bridge" between different ISAs. This is a powerful abstraction that bypasses the need for direct ISA-specific models.
  • Supervised Learning Advantage: While unsupervised methods exist (like UniMap), the paper argues that the availability of cross-architecture signals in binary analysis makes supervised learning feasible and potentially more effective for learning transferable embeddings.
  • Why Not IR? The authors investigate using Intermediate Representations (IR) like VEX as a bridge but find that even after lifting to a common IR, the resulting code can differ significantly in structure and length, making direct comparison difficult. This motivates their focus on learning instruction embeddings directly.
  • Data Scarcity is Solvable: The work demonstrates a practical solution to the data scarcity problem, enabling the reuse of models trained on abundant data to analyze low-resource ISAs, which is critical for comprehensive security analysis.

Key Results

  • Instruction Similarity Task (Intrinsic Evaluation):
    • CrossIns2Vec achieves AUC values of 0.78, 0.73, and 0.74 for x86↔ARM, x86↔MIPS, and x86↔PPC, respectively.
  • Compared to the baseline UniMap, CrossIns2Vec achieves higher AUCs (0.79, 0.71, 0.72 vs. 0.76, 0.66, 0.68 for the respective ISA pairs), indicating better capture of semantic relations.
  • Nearest neighbor analysis shows that x86 instructions have semantically similar counterparts in other ISAs, confirming the quality of learned embeddings.
  • Function Similarity Detection Task (Extrinsic Evaluation):
    • When trained on x86 and transferred, CrossIns2Vec achieves AUC values of 0.99 (ARM), 0.99 (MIPS), and 0.98 (PPC).
  • These results significantly outperform the baseline UniMap (AUCs of 0.95, 0.93, 0.94 for the respective ISAs), demonstrating superior transferability.
  • Malware Detection Task (Extrinsic Evaluation):
    • When trained on x86 and transferred, CrossIns2Vec achieves AUC values of 0.94 (ARM), 0.93 (MIPS), and 0.95 (PPC).
  • Again, these results are superior to the baseline UniMap (AUCs of 0.93, 0.91, 0.91), showcasing better transferability.
  • Table 4 shows that performance improves with larger training datasets, highlighting the importance of data quantity for the source ISA.

Limitations and Future Work

  • Generalizability: The authors acknowledge that the learned CAIE might not generalize to all types of code (e.g., Windows, iPhone, Android applications). Further investigation is needed.
  • Task Applicability: While the approach shows success in function similarity and malware detection, it's not claimed to be universally applicable to all binary analysis tasks. Future work could explore its application to other tasks.
  • Future Work: The paper suggests that much research can be done for exploring and expanding the boundaries of the approach. They also plan to release their datasets and models to facilitate further research.

Summary updated

BibTeX
@inproceedings{wang2024learning,
  author = {Junzhe Wang and Qiang Zeng and Lannan Luo},
  title = {{Learning Cross-Architecture Instruction Embeddings for Binary Code Analysis in Low-Resource Architectures}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.84},
}
Figure