Advising OpenMP Parallelization via A Graph-Based Approach with Transformers

Tal Kadosh, Nadav Schneider, Niranjan Hasabnis, Timothy G. Mattson, Yuval Pinter, Gal Oren

Abstract

As multi-core architectures become ubiquitous, the demand for efficient parallel programming has intensified. OpenMP remains the industry standard for shared-memory parallelization; however, manual implementation is error-prone and requires significant expertise. This paper introduces OMPify, a deep-learning-based tool designed to automate OpenMP pragma recommendation. By representing source code as a structured graph and utilizing a multi-label classification framework, OMPify achieves up to 90% accuracy on standard benchmarks, significantly outperforming existing specialized models and general-purpose large language models (LLMs).

1. Introduction

Writing parallel code is no longer a niche requirement but a necessity for modern performance. OpenMP simplifies this by allowing developers to insert compiler directives, or "pragmas," to manage thread distribution. Despite this, the manual insertion of pragmas is notoriously difficult. Incorrect directives can lead to race conditions, deadlocks, or degraded performance.

Traditional source-to-source (S2S) compilers often struggle with the complexity and variability of real-world code. While recent AI-driven approaches have attempted to treat code as natural language, they frequently overlook the underlying structural logic and data dependencies. OMPify addresses these gaps by treating code as a Data Flow Graph (DFG), enabling the model to understand the functional intent and variable relationships within C/C++ programs.

2. Methodology

OMPify is built upon a multimodal Transformer-based architecture, specifically leveraging GraphCodeBERT. Its design incorporates three primary innovations:

2.1 Graph-Based Representation

Unlike traditional models that process code as a linear sequence of tokens, OMPify utilizes a Data Flow Graph. This representation allows the model to track how variables interact across different scopes. Identifying these dependencies is critical for determining whether a loop can be safely parallelized and which clauses (such as private or reduction) are required.

2.2 Multi-Label Classification

Previous state-of-the-art tools, such as PragFormer, treated the prediction of pragmas and their associated clauses as independent binary tasks. OMPify adopts a multi-label approach, recognizing that the presence of a parallel pragma and its specific clauses are deeply interdependent. This unified prediction strategy ensures that the recommended pragmas are contextually consistent.

2.3 Curriculum Learning and Data Augmentation

To enhance robustness, the model was trained using a curriculum learning strategy. Training began with clean, well-structured code and progressively introduced "noise" through data augmentation techniques, such as variable renaming. This forced the model to prioritize the underlying logic and data flow over specific naming conventions.

3. Experimental Setup

The researchers developed a new dataset, Open-OMP-Plus, to train and validate the model. This dataset consists of over 54,000 C and C++ code snippets mined from GitHub, pairing serial loops with their corresponding "ground truth" OpenMP implementations. For evaluation, OMPify was tested against three classic high-performance computing (HPC) benchmarks: NAS, SPEC, and PolyBench.

4. Key Results and Insights

OMPify established a new state-of-the-art for automated parallelization advice, demonstrating several key findings:

  • Superior Accuracy: The model achieved up to 90% accuracy across the NAS, SPEC, and PolyBench benchmarks, consistently outperforming PragFormer in precision, recall, and F1-score.
  • The Importance of Scope: The study found that "Extended Scope"—analyzing code outside the immediate loop where variables are initialized—significantly improved the model's ability to correctly identify private and reduction clauses.
  • Structure vs. Text: Results confirmed that the "graphness" of code is a better predictor of parallelizability than its textual representation. Models that ignore structure are prone to "hallucinating" parallel opportunities where data dependencies actually prohibit them.
  • Comparison with LLMs: While general-purpose models like ChatGPT show high recall, they often suffer from low precision, suggesting pragmas in inapplicable scenarios. OMPify proved more disciplined, providing technically sound advice tailored to the constraints of the code.

5. Conclusion and Future Work

OMPify represents a significant step forward in making parallel programming more accessible. By combining graph-based code representation with multi-label Transformers, it provides a robust framework for automated OpenMP advice.

Future research will focus on expanding the model's capabilities in three areas:

  1. Pre-training: Utilizing a dedicated C/C++ corpus rather than general code datasets to improve language-specific nuances.
  2. Generative Modeling: Moving from classification to a generative approach to allow for the creation of more complex, custom pragma strings.
  3. Verification: Integrating formal verification tools to ensure that the suggested parallelization is not only likely to be correct but is mathematically proven to be thread-safe.

Summary updated

BibTeX
@inproceedings{kadosh2023advising,
  author = {Tal Kadosh and Nadav Schneider and Niranjan Hasabnis and Timothy G. Mattson and Yuval Pinter and Gal Oren},
  title = {{Advising OpenMP Parallelization via A Graph-Based Approach with Transformers}},
  booktitle = {IWOMP},
  year = {2023},
  doi = {10.1007/978-3-031-40744-4\_1},
}
Figure