MTFuzz: Fuzzing with a Multi-Task Neural Network

Dongdong She, Rahul Krishna, Lu Yan, Suman Jana, Baishakhi RayESEC/SIGSOFT FSE 2020 · DOI · OpenReview

Abstract

Fuzzing is a widely used technique for detecting software bugs and vulnerabilities. Most popular fuzzers generate new inputs using an evolutionary search to maximize code coverage. Essentially, these fuzzers start with a set of seed inputs, mutate them to generate new inputs, and identify the promising inputs using an evolutionary fitness function for further mutation. Despite their success, evolutionary fuzzers tend to get stuck in long sequences of unproductive mutations. In recent years, machine learning (ML) based mutation strategies have reported promising results. However, the existing ML-based fuzzers are limited by the lack of quality and diversity of the training data. As the input space of the target programs is high dimensional and sparse, it is prohibitively expensive to collect many diverse samples demonstrating successful and unsuccessful mutations to train the model. In this paper, we address these issues by using a Multi-Task Neural Network that can learn a compact embedding of the input space based on diverse training samples for multiple related tasks (i.e., predicting for different types of coverage). The compact embedding can guide the mutation process by focusing most of the mutations on the parts of the embedding where the gradient is high. MTFuzz uncovers 11 previously unseen bugs and achieves an average of 2× more edge coverage compared with 5 state-of-the-art fuzzers on 10 real-world programs.

Introduction

Software security relies heavily on fuzzing to identify vulnerabilities before they can be exploited. Modern "greybox" fuzzers typically employ evolutionary strategies, maintaining a corpus of "seed" inputs that are iteratively mutated. If a mutation reaches a previously unexplored part of the program's control-flow graph (CFG), it is saved as a new seed. However, this process often hits a plateau. Fuzzers frequently get stuck in long sequences of unproductive mutations because they lack a semantic understanding of which input bytes actually control the program's logic.

Recent research has turned to Machine Learning (ML) to guide these mutations. By training models to predict program behavior (like code coverage) based on input bytes, fuzzers can identify "hot-bytes"—specific locations in the input file most likely to trigger new behavior. However, these ML-guided approaches face a significant hurdle: data sparsity. In high-dimensional inputs, such as a 10KB binary file, only a tiny fraction of possible mutations will result in meaningful changes to program execution. Collecting enough diverse training data to teach a model where these critical bytes are is computationally expensive and often impractical.

The MTFuzz Methodology

MTFuzz addresses the data sparsity problem by leveraging Multi-Task Learning. Instead of training a model on a single metric, MTFuzz uses a Multi-Task Neural Network (MTNN) that learns from several related types of program feedback simultaneously. This approach allows the model to build a more robust internal representation of the input space, even when data for a specific task is limited.

The three-stage workflow of MTFuzz: training, guided mutation, and seed selection.

As shown in the workflow above, MTFuzz operates in three distinct stages:

  1. Stage 1: Train Multitask Feedforward NN: The fuzzer collects execution data from an initial seed corpus and trains the MTNN to predict multiple coverage metrics.
  2. Stage 2: Guided Mutation: The trained model is used to identify "hot-bytes" through saliency analysis (calculating gradients of the output with respect to the input). Mutations are then focused on these high-impact regions.
  3. Stage 3: Seed Selection and Incremental Learning: New inputs that achieve novel coverage are added to the seed corpus, and the model is periodically retrained to adapt to the newly discovered program logic.

Multi-Task Neural Network Architecture

The technical core of MTFuzz is its MTNN architecture, which consists of shared encoding layers and multiple task-specific decoding layers.

Detailed architecture of the Multi-Task Neural Network (MTNN) used in MTFuzz.

By forcing a single encoder to satisfy three different tasks, the model is pressured to learn a compact embedding ($z$) that captures the most essential semantic features of the input. The three tasks utilized are:

  • Edge Coverage (Primary): Predicting which control-flow edges are triggered by an input.
  • Approach-Sensitive Coverage (Auxiliary): Measuring how "close" an input got to reaching an edge that was not hit.
  • Context-Sensitive Coverage (Auxiliary): Identifying the function call stack context from which an edge was reached.

To further improve performance on difficult-to-reach code, MTFuzz implements an Adaptive Loss function. This function penalizes false negatives more heavily for "rare edges," ensuring the model does not ignore infrequent but critical program behaviors as mere noise.

Key Results and Insights

The authors evaluated MTFuzz against five state-of-the-art fuzzers (AFL, AFLFast, FairFuzz, Angora, and Neuzz) across 10 real-world programs, including readelf, libjpeg, and libxml2.

  • Bug Discovery: MTFuzz successfully identified 71 real-world bugs, 11 of which were previously unknown vulnerabilities, including heap overflows and memory leaks.
  • Superior Coverage: MTFuzz achieved an average of 2× more edge coverage than Neuzz (the leading ML-based fuzzer at the time) and significantly outperformed traditional evolutionary fuzzers.
  • Transfer Learning: A significant finding was the effectiveness of transfer learning. By transferring the compact embedding learned from one program (e.g., one XML parser) to another similar program, MTFuzz achieved up to 14× more coverage compared to starting the learning process from scratch.
  • Model Accuracy: The inclusion of auxiliary tasks and adaptive loss improved the model's recall for rare edges by approximately 15%, directly translating to better discovery of deep program logic.

Conclusion

MTFuzz demonstrates that multi-task learning can effectively overcome the data sparsity challenges inherent in ML-guided fuzzing. By learning a compact, multi-purpose embedding of the input space, MTFuzz identifies critical input regions more accurately than single-task models. While the system currently requires an initial "blind" fuzzing phase to bootstrap training data, its ability to discover 11 new bugs in widely-used software highlights the potential of neural-guided security tools. Future work may involve applying this multi-task approach to even more complex targets, such as operating system kernels or web browsers.

Summary updated

Abstract

Fuzzing is a widely used technique for detecting software bugs and vulnerabilities. Most popular fuzzers generate new inputs using an evolutionary search to maximize code coverage. Essentially, these fuzzers start with a set of seed inputs, mutate them to generate new inputs, and identify the promising inputs using an evolutionary fitness function for further mutation. Despite their success, evolutionary fuzzers tend to get stuck in long sequences of unproductive mutations. In recent years, machine learning (ML) based mutation strategies have reported promising results. However, the existing ML-based fuzzers are limited by the lack of quality and diversity of the training data. As the input space of the target programs is high dimensional and sparse, it is prohibitively expensive to collect many diverse samples demonstrating successful and unsuccessful mutations to train the model. In this paper, we address these issues by using a Multi-Task Neural Network that can learn a compact embedding of the input space based on diverse training samples for multiple related tasks (i.e., predicting for different types of coverage). The compact embedding can guide the mutation process by focusing most of the mutations on the parts of the embedding where the gradient is high. \tool uncovers $11$ previously unseen bugs and achieves an average of $2\times$ more edge coverage compared with 5 state-of-the-art fuzzer on 10 real-world programs.
BibTeX
@inproceedings{she2020mtfuzz,
  author = {Dongdong She and Rahul Krishna and Lu Yan and Suman Jana and Baishakhi Ray},
  title = {{MTFuzz: Fuzzing with a Multi-Task Neural Network}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2020},
  doi = {10.1145/3368089.3409723},
}
Figure