MPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers

Nadav Schneider, Tal Kadosh, Niranjan Hasabnis, Timothy G. Mattson, Yuval Pinter, Gal Oren

Introduction

In high-performance computing (HPC), scientific simulations frequently exceed the memory and processing capabilities of individual compute nodes. To address this, developers utilize the Message Passing Interface (MPI) to implement distributed-memory parallelism through domain decomposition. Unlike shared-memory models such as OpenMP, which are relatively intuitive, MPI requires explicit management of data dependencies and inter-processor communication. The complexity of manual MPI implementation often leads to deadlocks or race conditions, creating a significant barrier for developers. While automated tools for shared-memory parallelization are well-established, data-driven assistance for distributed-memory MPI programming has remained a critical gap in the field.

Methodology: MPI-RICAL

To bridge this gap, we present MPI-RICAL, a programming assistance tool built upon the SPT-Code architecture. SPT-Code is a specialized Transformer model designed for semantic understanding across multiple programming languages.

Translation-Based Parallelization

MPI-RICAL frames the task of MPI parallelization as a code translation problem. The model accepts serial code (or code with MPI calls stripped) as input and learns to "translate" it into a valid MPI-parallelized version. This approach allows the model to learn the underlying relationship between serial computational logic and the structural requirements of distributed communication.

Structural Representation (X-SBT)

A key technical innovation in MPI-RICAL is the use of the X-SBT representation. Rather than processing raw source code or high-overhead Abstract Syntax Trees (ASTs), the model utilizes a linearized, XML-like format. This representation preserves the structural logic of the code while maintaining a sequence length manageable for Transformer architectures.

Prediction Objectives

The model is optimized to address two primary research questions during the parallelization process:

  1. Function Selection: What specific MPI function is required?
  2. Placement Optimization: Where should the function be inserted within the code block?

Data Collection and Benchmarking

The effectiveness of MPI-RICAL is supported by the introduction of MPICodeCorpus, the first large-scale public corpus of MPI-based parallel programs.

  • Corpus Construction: The dataset was curated by mining over 15,000 open-source GitHub repositories, resulting in approximately 25,000 C programs after rigorous filtering for parsing success and length constraints.
  • Numerical Benchmark: To ensure the model generalizes to real-world scientific logic, a secondary benchmark of 11 fully compiled numerical programs was developed. These programs cover essential algorithms such as Matrix-Vector Multiplication, Merge Sort, and Monte-Carlo simulations for Pi calculation.

Results and Analysis

MPI-RICAL demonstrates robust performance across both general and specialized datasets, suggesting that Transformers can effectively capture the nuances of distributed parallelism.

Performance Metrics

On the MPICodeCorpus test set, the model achieved an F1 score of 0.87 and a BLEU score of 0.93, indicating high structural fidelity to human-authored code. In the specialized numerical benchmark, the model reached a Precision of 0.98 and an M-F1 score of 0.91. The model was particularly successful with algorithms like Fibonacci and the Trapezoidal Rule, achieving perfect 1.0 scores in those categories.

The "Common Core" and Tolerance

The research identified a "Common Core" of MPI functions (e.g., MPI_Init, MPI_Comm_rank, MPI_Finalize) that follow a power-law distribution. Mastery of these core functions is central to the model's success. Furthermore, the evaluation employed a "one-line tolerance" metric, recognizing that placing a call one line away from the ground truth often maintains program validity in a practical development environment.

Limitations and Future Directions

While MPI-RICAL represents a significant step forward, several limitations remain:

  • Context Window: Current hardware constraints limit the model to approximately 50 lines of code (320 tokens), hindering its application to large-scale, multi-file projects.
  • Argument Generation: The current iteration suggests function names and locations but does not yet populate specific arguments (e.g., buffer pointers or communicators).
  • Language Specialization: While fine-tuned on C, the model's pre-training on Java and Python suggests that a C/C++ specific pre-training phase could further enhance accuracy.

Future work will focus on integrating MPI-RICAL into Integrated Development Environments (IDEs) like VS Code, providing real-time "Copilot" capabilities tailored specifically for the HPC community.

Summary updated

BibTeX
@inproceedings{schneider2023mpi,
  author = {Nadav Schneider and Tal Kadosh and Niranjan Hasabnis and Timothy G. Mattson and Yuval Pinter and Gal Oren},
  title = {{MPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers}},
  booktitle = {Workshops @ SC},
  year = {2023},
  doi = {10.1145/3624062.3624063},
}
Figure