DataRaceBench V1.4.1 and DataRaceBench-ML V0.1: Benchmark Suites for Data Race Detection

Le Chen, Wenhao Wu, Stephen F. Siegel, Pei-Hung Lin, Chunhua LiaoarXiv 2023 · OpenReview

Abstract

Data races pose a significant threat in multi-threaded parallel applications due to their negative impact on program correctness. DataRaceBench, an open-source benchmark suite, is specifically crafted to assess these data race detection tools in a systematic and measurable manner. Machine learning techniques have recently demonstrated considerable potential in high-performance computing (HPC) program analysis and optimization. However, these techniques require specialized data formats for training and refinement. This paper presents the latest update to DataRaceBench, incorporating new data race contributions from Wu et al., and introduces a derived dataset named DataRaceBench-ML (DRB-ML). DRB-ML aligns with the emerging trend of machine learning and large language models. Originating from DataRaceBench, this dataset includes detailed labels that denote the presence of a data race and provides comprehensive details of associated variables, such as variable names, line numbers, and the operation (read/write). Unique to DRB-ML, we have also integrated a series of tailored prompt-response pairs specifically designed for LLM fine-tuning.

Introduction

Data races—occurring when multiple threads access the same memory location simultaneously with at least one access being a write—are among the most elusive bugs in parallel programming. Their non-deterministic nature makes them notoriously difficult to reproduce and debug. While traditional static and dynamic detectors have been evaluated using suites like DataRaceBench, the research community is increasingly turning toward Machine Learning (ML) and Large Language Models (LLMs) for automated bug detection and localization.

However, a significant bottleneck exists: AI models require structured, high-quality data to learn effectively. Most existing HPC benchmarks consist of raw source code, which lacks the granular labeling needed for a model to not only identify the existence of a race but also locate its exact coordinates (variable, line, and operation). This work addresses this gap by expanding the standard DataRaceBench suite and introducing DataRaceBench-ML, a dataset specifically formatted for modern AI-driven code analysis.

Methodology and Data Transformation

The development of these suites involved a multi-step pipeline designed to modernize the benchmarks for both traditional and AI-based tools.

Expansion of DataRaceBench

The suite was first expanded to version 1.4.1 by integrating 20 new C-language microbenchmarks derived from recent formal verification studies. This brings the total to 201 cases, each meticulously categorized as "race-yes" or "race-no." These cases are designed to be minimal and self-contained, isolating specific concurrency patterns.

The DRB-ML Pipeline

To create DataRaceBench-ML, the researchers developed an automated transformation pipeline to convert raw microbenchmarks into a structured format suitable for ML training.

Workflow for generating the DataRaceBench-ML dataset from raw microbenchmarks.

As illustrated in the workflow, the process involves several key steps:

  1. Comment Trimming: Source code is stripped of comments to prevent models from "cheating" by reading developer notes or labels embedded in the code.
  2. Metadata Extraction: Scripts extract precise variable names, line numbers, and column offsets for the instructions involved in the race.
  3. Structured Output: The data is packaged into JSON format, including keys for variable pairs, operation types (read/write), and code length.
  4. Instruction Tuning Generation: For LLM applications, the pipeline generates prompt-response pairs where the prompt asks the model to act as an HPC expert, and the response provides the ground-truth analysis.

Dataset Characteristics

DataRaceBench (DRB) v1.4.1

DRB remains the industry standard for evaluating traditional detectors. It primarily focuses on OpenMP and C/C++ microbenchmarks, providing a rigorous testbed for tool accuracy.

DataRaceBench-ML (DRB-ML) v0.1

DRB-ML is a derivative dataset that enables more advanced ML tasks:

  • Localization: Unlike binary classification datasets, DRB-ML provides the "why" and "where," allowing models to learn how to point to specific conflicting memory accesses.
  • Trimmed Code Strategy: By using trimmed_code, the dataset ensures that models learn from the logic of the code rather than external documentation.
  • LLM-Ready Formatting: The inclusion of prompt-response pairs facilitates supervised fine-tuning (SFT) for models like Llama or StarCoder, aligning them with the specific domain of HPC concurrency.

Key Insights and Results

The expansion to 201 microbenchmarks provides a more diverse set of race conditions. For example, DRB193 demonstrates a race caused by mismatched named critical sections (critical(A) vs critical(B)). DRB-ML successfully maps this complex logic to specific line numbers in the trimmed code, providing the necessary ground truth for a model to learn this specific synchronization failure.

Conclusion and Future Work

While the current suite of 201 microbenchmarks is a significant step forward, it remains relatively small for training deep learning models from scratch. Future efforts will focus on scaling these datasets to include larger, multi-file HPC applications where data races are often obscured by architectural complexity. Additionally, the authors plan to evaluate various LLMs using DRB-ML to establish a baseline for AI performance against traditional detection tools. By bridging the gap between HPC correctness and modern AI, DataRaceBench-ML provides a foundation for the next generation of automated debugging tools.

Summary updated

Abstract

Data races pose a significant threat in multi-threaded parallel applications due to their negative impact on program correctness. DataRaceBench, an open-source benchmark suite, is specifically crafted to assess these data race detection tools in a systematic and measurable manner. Machine learning techniques have recently demonstrated considerable potential in high-performance computing (HPC) program analysis and optimization. However, these techniques require specialized data formats for training and refinement. This paper presents the latest update to DataRaceBench, incorporating new data race contributions from Wu et al. \cite{wu2023model}, and introduces a derived dataset named DataRaceBench-ML (DRB-ML) \cite{drbml}. DRB-ML aligns with the emerging trend of machine learning and large language models. Originating from DataRaceBench, this dataset includes detailed labels that denote the presence of a data race and provides comprehensive details of associated variables, such as variable names, line numbers, and the operation (read/write). Unique to DRB-ML, we have also integrated a series of tailored prompt-response pairs specifically designed for LLM fine-tuning.
BibTeX
@misc{chen2023dataracebench,
  author = {Le Chen and Wenhao Wu and Stephen F. Siegel and Pei-Hung Lin and Chunhua Liao},
  title = {{DataRaceBench V1.4.1 and DataRaceBench-ML V0.1: Benchmark Suites for Data Race Detection}},
  year = {2023},
  eprint = {2308.08473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.08473},
}
Figure