GPU acceleration of non-equilibrium Green's function calculation using OpenACC and CUDA FORTRAN

Chao Yang, Jia Yin, Khaled Z. Ibrahim, Mauro Del Ben, Jack Deslippe, Yang-hao Chan, Chao Yang 0001arXiv 2025 · DOI · arXiv

Introduction

Understanding the dynamics of quantum many-body systems—such as electron movement in semiconductors following laser excitation—requires solving the Kadanoff-Baym equations (KBEs). These equations yield Non-Equilibrium Green’s Functions (NEGFs), which describe particle propagation through space and time.

However, solving KBEs is computationally intensive due to the "curse of dimensionality." The functions must be propagated on a two-time grid ($t, t'$), leading to a computational cost that grows cubically ($O(N^3)$) with simulation time. Historically, this bottleneck has restricted research to small systems or short timescales, as even a 10-step simulation on a modern CPU can exceed 24 hours.

The two-time grid (t, t') used for propagating Non-Equilibrium Green’s Functions.

GPU-Accelerated Methodology

To overcome these limitations, researchers developed highly optimized GPU-accelerated solvers using two primary approaches:

  1. OpenACC: This directive-based approach allowed for relatively straightforward implementation. By utilizing advanced features like loop gang and loop vector for thread hierarchy management and collapse clauses for loop fusion, the team ensured high utilization of the GPU's parallel architecture.
  2. CUDA FORTRAN: For deeper optimization, CUDA FORTRAN provided low-level control over 3D thread blocks and grids. This enabled "task parallelism," allowing simultaneous calculation of different self-energy components (the $P$ and $\Sigma^2$ terms) within a single kernel.

Key Optimization Strategies

  • Loop Reordering: Intensive loops were moved to the innermost levels to maximize data reuse.
  • Memory Management: Allocations were moved outside main time-stepping loops to minimize kernel launch overhead and CPU-GPU data transfer.
  • On-the-fly Indexing: Instead of pre-computed array lookups, indices were calculated mathematically to ensure coalesced memory access, significantly improving throughput.

Performance Results

The transition to GPU acceleration yielded transformative results. For the most computationally expensive component—self-energy calculations—the GPU implementations achieved speedups of 107x to 126x over traditional CPU methods. Collision integrals also saw significant improvements, with speedups ranging from 15x to 19x.

The impact of incremental optimizations is particularly striking. A naive OpenACC implementation initially required 181.61 seconds for a specific test run. Through a combination of loop fusion, single-kernel restructuring, and on-the-fly indexing, this was reduced to just 2.31 seconds.

Impact of successive GPU optimizations on execution time, reducing a test run from 181.61s to 2.31s.

Scalability and Future Directions

The solver demonstrated excellent "weak scaling," indicating its capability to handle larger physical systems as hardware resources increase. However, "strong scaling" faces challenges; as the number of MPI ranks exceeds 128, communication overhead (specifically MPI_Allreduce) begins to limit performance gains.

Future work aims to extend these optimizations beyond the "second-Born approximation" to more complex physics, such as the GW approximation. Additionally, researchers are exploring "out-of-core" memory management strategies to handle two-time grids that exceed the physical memory limits of current GPUs.

Summary updated

BibTeX
@misc{yang2025gpu,
  author = {Chao Yang and Jia Yin and Khaled Z. Ibrahim and Mauro Del Ben and Jack Deslippe and Yang-hao Chan and Chao Yang},
  title = {{GPU acceleration of non-equilibrium Green's function calculation using OpenACC and CUDA FORTRAN}},
  year = {2025},
  eprint = {2505.19467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19467},
}
Figure