As multi-core and heterogeneous hardware architectures become ubiquitous, software performance increasingly depends on parallel execution. However, writing parallel code remains notoriously difficult. Developers must manually manage complex abstractions like thread hierarchies, memory synchronization, and data partitioning. This manual concurrency management is highly error-prone, frequently introducing non-deterministic bugs such as data races, deadlocks, and performance regressions.
To bridge this gap between high-level application logic and low-level hardware complexity, researchers have developed techniques to automatically transform sequential programs into efficient parallel implementations. By shifting the burden of concurrency management from the programmer to automated tools, this research direction aims to unlock hardware parallelism while preserving the simplicity of sequential programming.
Rule-Based and Semantic Refactoring
The earliest automated approaches rely on formal source-to-source refactoring. The core insight is that many sequential code patterns—particularly imperative loops—can be mapped directly to well-defined parallel structures if their data dependencies allow it.
To do this safely, an automated tool must analyze the data flow of a program to ensure that executing tasks concurrently will not alter the program's final output. For example, Junfeng Zhao developed a tool called JMRT in Refactoring Java Code to MapReduce Framework (S) that automatically refactors sequential Java loops into the MapReduce paradigm for Apache Spark. Instead of forcing the developer to reason about distributed data dependencies, JMRT converts imperative loops into a functional intermediate representation. Because functional representations eliminate side effects and make data dependencies explicit, the tool can safely apply mathematical transformation rules to generate parallel code.
[Sequential Imperative Loop]
│
▼ (Explicit Side-Effects & State)
[Functional Intermediate Representation]
│
▼ (Formal Pattern Rewrite Rules)
[Parallel MapReduce / OpenMP Code]
Similarly, Christopher Brown 0002 et al. introduced the PARAPHRASE tool in Generating Parallel Programs Using Refactoring, which utilizes a system of formal pattern rewrite rules to automatically restructure sequential code into parallel patterns.
A major challenge in rule-based refactoring is handling high-level language abstractions. Traditional parallelizing compilers are often restricted to primitive data types and simple arrays. To address this, Chunhua Liao et al. demonstrated in Extending Automatic Parallelization to Optimize High-Level Abstractions for Multicore that semantic-aware parallelizers can recognize high-level C++ abstractions, such as Standard Template Library (STL) containers. By embedding knowledge of container semantics directly into the compiler infrastructure, the system identifies parallelization opportunities that would otherwise be obscured by the complexity of pointer-heavy object code.
Speculative and Dynamic Parallelization
Static analysis frequently fails when data dependencies are irregular, pointer-heavy, or dependent on runtime inputs. When a compiler cannot definitively prove that two operations are independent, it must choose the conservative path and execute them sequentially.
To overcome this limitation, researchers use Thread-Level Speculation (TLS) and dynamic task graphs. Instead of proving independence before execution, speculative systems run code segments in parallel under the assumption that dependencies will not conflict. If a conflict is detected at runtime, the system rolls back the speculative execution to a safe state.
In The Jrpm system for dynamically parallelizing Java programs, Michael K. Chen designed a system that combines dynamic compilation with hardware profiling to automatically parallelize loops via TLS. By continuously monitoring memory access patterns at runtime, the system can parallelize loops that static compilers would flag as unsafe due to potential pointer aliasing.
On the compiler side, Josep Torrellas et al. developed POSH in POSH: a TLS compiler that exploits program structure. POSH uses existing program structures, such as subroutines and loop iterations, to automatically generate speculative tasks, using a profiling pass to discard tasks that frequently trigger rollbacks.
Alternatively, runtime libraries can build dynamic dependency graphs to orchestrate execution safely without full hardware speculation. For instance, the Parsl library, presented by Kyle Chard et al. in Extended Abstract, uses simple Python decorators to intercept function calls. As the program runs, Parsl automatically constructs a dynamic dependency graph based on shared inputs and outputs, executing tasks asynchronously as soon as their data prerequisites are met.
Learning-Based Parallelization and Translation
The rise of deep learning has introduced a data-driven paradigm to automatic parallelization. Traditional compilers rely on rigid, deterministic heuristics that struggle with the messy pragmatics of real-world source code. Machine learning models, by contrast, can learn to identify parallelizable regions and predict synchronization requirements by training on massive code repositories.
This shift is particularly evident in the classification of parallelizable loops. In PragFormer: Data-Driven Parallel Source Code Classification with Transformers, Re'em Harel et al. introduced a Transformer-based model trained specifically on OpenMP source code. PragFormer analyzes sequential code to determine not only whether a loop can be parallelized, but also what specific data-sharing clauses (such as private or shared) are required to prevent data races.
To capture both the semantic meaning and the structural syntax of code, hybrid architectures combine graph structures with language models. Ali TehraniJamsaz et al. developed AUTOPARLLM in AUTOPARLLM: GNN-Guided Automatic Code Parallelization using Large Language Models. This framework uses a Graph Neural Network (GNN) to analyze the control flow and data dependencies of a program, generating a structural context vector. This vector is then fed into a Large Language Model (LLM), guiding it to insert correct OpenMP directives into sequential code far more reliably than a vanilla LLM could on its own.
┌─── Control Flow Graph ───┐
│ │
Sequential Code ──┼─── Data Dependency ──────┼─► GNN ─► Structural Context ──┐
│ │ │
└─── Abstract Syntax Tree ─┘ ▼
Large Language Model ─► Parallel Code
Beyond adding directives to existing code, deep learning models can automate full language-to-language translation, such as migrating sequential C code to specialized GPU architectures like CUDA. Yuanbo Wen et al. introduced BabelTower in BabelTower: Learning to Auto-parallelized Program Translation, which leverages unsupervised back-translation and a parallel-aware reranker to translate sequential C code into high-performance CUDA code.
To guarantee that these learning-based translations remain functionally correct, Changxin Ke et al. proposed QiMeng-MuPa in QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translation. This framework utilizes an iterative "Translator-Tester" loop. The translator generates the parallel code, while the tester verifies functional equivalence against the original sequential execution. The feedback from the tester is used to refine the translator, ensuring that the automated system optimizes performance without sacrificing semantic correctness.