Implementing CUDA Streams into AstroAccelerate -- A Case Study

Jan Novotný, Karel Adámek, Wesley ArmourarXiv 2021 · DOI · arXiv

Abstract To be able to run tasks asynchronously on NVIDIA GPUs, a programmer must explicitly implement asynchronous execution in their code using the syntax of CUDA streams. Streams allow a programmer to launch independent concurrent execution tasks, providing the ability to utilize different functional units on the GPU asynchronously. For example, it is possible to transfer the results from a previous computation performed on input data $n-1$ over the PCIe bus while computing the result for input data $n$ by placing different tasks in different CUDA streams. The benefit of such an approach is that the time taken for the data transfer between the host and device can be hidden with computation. This case study deals with the implementation of CUDA streams into AstroAccelerate, a GPU-accelerated real-time signal processing pipeline for time-domain radio astronomy.

Introduction

In radio astronomy, "real-time" processing is a technical necessity. Next-generation instruments like the Square Kilometre Array (SKA) generate data at rates so high that storing raw data for later analysis is unfeasible; it must be processed as it arrives. While GPUs provide the massive parallel compute power required for tasks like dedispersion, they often suffer from idle periods while waiting for data to travel across the PCIe bus. Standard synchronous code executes linearly: transfer data, run the kernel, and transfer results back. This paper explores how implementing asynchronous execution via CUDA streams can eliminate these idle gaps and maximize hardware utilization.

Methods

The authors integrated CUDA streams into the AstroAccelerate (AA) framework using a multi-pronged technical approach:

  1. Data Chunking: Input signals were divided into large "time chunks" fitting GPU memory, which were then subdivided into smaller chunks assigned to specific CUDA stream IDs.
  2. Pinned Memory: The team moved from standard pageable host memory to explicit pinned (page-locked) memory allocation using cudaMallocHost. This allows the GPU to access host memory directly, enabling faster transfers and overlapping.
  3. Asynchronous API Calls: Synchronous cudaMemcpy calls were replaced with cudaMemcpyAsync, allowing the CPU to issue commands and immediately proceed to subsequent tasks.
  4. Multi-threading with OpenMP: To prevent host-to-host memory copies from blocking CUDA streams, the authors used OpenMP to create dedicated CPU threads for each stream ID, ensuring true concurrency between the host and device.
Comparison of synchronous execution in a single stream versus asynchronous execution across multiple streams.

Key Insights

The "Hidden" Driver Cost

Visualizing performance in the NVIDIA Visual Profiler revealed significant "timing gaps" when using pageable memory. These gaps represent the driver's overhead in creating temporary pinned buffers. Explicitly pinning memory removes this overhead, smoothing the timeline and increasing efficiency.

Concurrency and Resource Saturation

Kernel overlap is resource-dependent. If a single kernel fully saturates the GPU's streaming multiprocessors (SMs), concurrent kernels cannot run. Overlap typically occurs at the tail end of a kernel's execution as resources are gradually released.

Results

The implementation was evaluated using a simulated SKA-like input signal (1400 MHz central frequency, 300 MHz bandwidth, 4096 channels).

  • Overall Speedup: The stream-enabled version of AstroAccelerate achieved a 1.33x speedup.
  • Throughput: Memory transfer throughput increased by approximately 30%.
  • Visual Verification: Profiler traces confirmed that Host-to-Device (H2D) transfers, kernel executions, and Device-to-Host (D2H) transfers were successfully overlapping, effectively "hiding" the PCIe latency behind computation.
Profiler traces comparing execution with driver overhead (top) versus optimized asynchronous execution (bottom).

Limitations and Future Work

While the dedispersion phase saw significant gains, the single pulse search phase remains synchronous. The primary obstacle is the use of atomic operations; the authors found that cudaStreamWaitEvent()—essential for stream synchronization—does not interact reliably with atomic operations in the current pipeline. Future work will focus on refactoring these kernels to support atomics in an asynchronous environment, aiming to extend the 1.33x speedup across the entire processing pipeline.

Summary updated

Abstract

To be able to run tasks asynchronously on NVIDIA GPUs a programmer must explicitly implement asynchronous execution in their code using the syntax of CUDA streams. Streams allow a programmer to launch independent concurrent execution tasks, providing the ability to utilise different functional units on the GPU asynchronously. For example, it is possible to transfer the results from a previous computation performed on input data n-1, over the PCIe bus whilst computing the result for input data n, by placing different tasks in different CUDA streams. The benefit of such an approach is that the time taken for the data transfer between the host and device can be hidden with computation. This case study deals with the implementation of CUDA streams into AstroAccelerate. AstroAccelerate is a GPU accelerated real-time signal processing pipeline for time-domain radio astronomy.
BibTeX
@misc{novotny2021implementing,
  author = {Jan Novotný and Karel Adámek and Wesley Armour},
  title = {{Implementing CUDA Streams into AstroAccelerate -- A Case Study}},
  year = {2021},
  eprint = {2101.00941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.00941},
}
Figure