Shellcode_IA32: A Dataset for Automatic Shellcode Generation

Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, Samira ShaikhACL 2021

TL;DR

This paper introduces Shellcode_IA32, the first large-scale dataset for automatically generating x86 shellcode from natural language descriptions. By framing shellcode generation as a machine translation task, they establish baseline performance using Neural Machine Translation (NMT) models, paving the way for future research in this critical area of cybersecurity.

What Problem Does This Paper Address?

Generating shellcode, the small pieces of code used to exploit software vulnerabilities, is a technically challenging task typically requiring deep expertise in assembly language. While automated code generation from natural language has seen progress in general programming languages, it has remained largely unexplored for shellcode. This gap is significant because shellcode is a fundamental component of cyberattacks, but also a valuable tool for security researchers to demonstrate vulnerabilities and develop defenses. The authors aim to bridge this gap by creating a dataset and evaluating initial machine translation approaches for this task.

Methods

The core of the paper's methodology is treating shellcode generation as a Neural Machine Translation (NMT) problem. They leverage a standard encoder-decoder architecture with attention, similar to those used in natural language translation.

  • Encoder: A bi-directional LSTM processes the natural language "intent" (comment) and transforms it into a vector representation.
  • Decoder: Another LSTM, guided by an attention mechanism, generates the corresponding x86 assembly code snippet.
  • Training: They use a train/dev/test split of their newly created dataset and employ an Adam optimizer.
  • Inference: A beam search strategy with a beam size of 5 is used for generating the output assembly code.

This approach is standard within the NMT field, but its application to the domain of low-level assembly shellcode is novel.

Datasets

The primary contribution is the introduction of the Shellcode_IA32 dataset.

  • Content: It comprises 3,200 lines of x86 assembly code extracted from real-world shellcodes found in public exploit databases (Shell-storm, Exploit Database) and supplemented with examples from assembly programming tutorials.
  • Format: Each entry consists of a natural language description (intent) paired with its corresponding assembly code snippet.
  • Key Features:
    • Includes challenging but common assembly instructions.
  • Features approximately 16% multi-line snippets, reflecting more complex generation tasks.
  • Focuses on Linux environments and uses Netwide Assembler (NASM) syntax.
  • Comparison: The dataset is comparable in size to other code generation datasets like CoNaLa but is unique in its focus on security-specific assembly code. The authors emphasize that unlike general code, shellcode cannot be easily mined from general web sources due to its specialized nature.

Key Insights and Ideas

  • Shellcode Generation as Machine Translation: Framing this security-critical task as a standard NLP problem opens up a new avenue for research and development.
  • Importance of Multi-line Snippets: The inclusion of multi-line snippets is crucial, as many shellcode operations require sequences of instructions, and treating them as single units is more realistic than a one-to-one line mapping.
  • Variability in Natural Language Descriptions: The authors acknowledge and account for the variability in how intents are described by having multiple authors contribute descriptions and by using developer comments where available.
  • Challenges in Implicit Knowledge: The model struggles with intents that rely on implicit knowledge (e.g., understanding that "inversion" implies a bitwise NOT operation) or require choosing from a large set of similar instructions (e.g., conditional jumps).

Key Results

  • Baseline Performance: The paper establishes baseline performance metrics using NMT.
  • BLEU Scores: Achieved scores ranging from 15.62% (BLEU-4) for a 2-layer, 64-dimension model to 76.12% (BLEU-4) for a 1-layer, 512-dimension model.
  • Exact Match Accuracy (ACC): Ranged from 9.38% to 51.25% for the same model configurations.
  • Hyperparameter Tuning: Experiments showed that increasing the number of layers generally led to worse performance, while layer dimensions between 256 and 512 yielded the best results.
  • Qualitative Analysis:
    • The model successfully translated some multi-line intents and correctly used specific addressing modes (e.g., lea with brackets).
  • Failures included generating incorrect instructions (e.g., and instead of not) and misinterpreting conditional jumps (e.g., jle instead of jge). These errors were often attributed to a lack of implicit knowledge or ambiguity in the intent.

Limitations and Future Work

  • Dataset Size: While comparable to some NLP datasets, the 3,200 instances might still be considered small for complex NMT tasks, especially given the specialized nature of shellcode.
  • Model Complexity: The best performance was achieved with a relatively simple 1-layer model, suggesting that more complex architectures might not always be beneficial for this specific task and dataset.
  • Implicit Knowledge Gap: The model's difficulty with intents requiring implicit knowledge highlights a significant challenge for future work.
  • Future Directions:
    • Expanding the dataset with more diverse shellcodes and intents.
  • Exploring more advanced NMT architectures and techniques tailored for low-level code generation.
  • Developing methods to better incorporate implicit domain knowledge into the models.
  • Investigating techniques to improve the accuracy of conditional jumps and other instruction choices.
  • Ultimately, moving towards generating more complex and functional shellcode beyond single instructions or short sequences.

Summary updated

BibTeX
@inproceedings{liguori2021shellcode,
  author = {Pietro Liguori and Erfan Al-Hossami and Domenico Cotroneo and Roberto Natella and Bojan Cukic and Samira Shaikh},
  title = {{Shellcode\_IA32: A Dataset for Automatic Shellcode Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure