Abstract To be able to run tasks asynchronously on NVIDIA GPUs, a programmer must explicitly implement asynchronous execution in their code using the syntax of CUDA streams. Streams allow a programmer to launch independent concurrent execution tasks, providing the ability to utilize different functional units on the GPU asynchronously. For example, it is possible to transfer the results from a previous computation performed on input data $n-1$ over the PCIe bus while computing the result for input data $n$ by placing different tasks in different CUDA streams. The benefit of such an approach is that the time taken for the data transfer between the host and device can be hidden with computation. This case study deals with the implementation of CUDA streams into AstroAccelerate, a GPU-accelerated real-time signal processing pipeline for time-domain radio astronomy.
Introduction
In radio astronomy, "real-time" processing is a technical necessity. Next-generation instruments like the Square Kilometre Array (SKA) generate data at rates so high that storing raw data for later analysis is unfeasible; it must be processed as it arrives. While GPUs provide the massive parallel compute power required for tasks like dedispersion, they often suffer from idle periods while waiting for data to travel across the PCIe bus. Standard synchronous code executes linearly: transfer data, run the kernel, and transfer results back. This paper explores how implementing asynchronous execution via CUDA streams can eliminate these idle gaps and maximize hardware utilization.
Methods
The authors integrated CUDA streams into the AstroAccelerate (AA) framework using a multi-pronged technical approach:
- Data Chunking: Input signals were divided into large "time chunks" fitting GPU memory, which were then subdivided into smaller chunks assigned to specific CUDA stream IDs.
- Pinned Memory: The team moved from standard pageable host memory to explicit pinned (page-locked) memory allocation using
cudaMallocHost. This allows the GPU to access host memory directly, enabling faster transfers and overlapping. - Asynchronous API Calls: Synchronous
cudaMemcpycalls were replaced withcudaMemcpyAsync, allowing the CPU to issue commands and immediately proceed to subsequent tasks. - Multi-threading with OpenMP: To prevent host-to-host memory copies from blocking CUDA streams, the authors used OpenMP to create dedicated CPU threads for each stream ID, ensuring true concurrency between the host and device.
Key Insights
The "Hidden" Driver Cost
Visualizing performance in the NVIDIA Visual Profiler revealed significant "timing gaps" when using pageable memory. These gaps represent the driver's overhead in creating temporary pinned buffers. Explicitly pinning memory removes this overhead, smoothing the timeline and increasing efficiency.
Concurrency and Resource Saturation
Kernel overlap is resource-dependent. If a single kernel fully saturates the GPU's streaming multiprocessors (SMs), concurrent kernels cannot run. Overlap typically occurs at the tail end of a kernel's execution as resources are gradually released.
Results
The implementation was evaluated using a simulated SKA-like input signal (1400 MHz central frequency, 300 MHz bandwidth, 4096 channels).
- Overall Speedup: The stream-enabled version of AstroAccelerate achieved a 1.33x speedup.
- Throughput: Memory transfer throughput increased by approximately 30%.
- Visual Verification: Profiler traces confirmed that Host-to-Device (H2D) transfers, kernel executions, and Device-to-Host (D2H) transfers were successfully overlapping, effectively "hiding" the PCIe latency behind computation.
Limitations and Future Work
While the dedispersion phase saw significant gains, the single pulse search phase remains synchronous. The primary obstacle is the use of atomic operations; the authors found that cudaStreamWaitEvent()—essential for stream synchronization—does not interact reliably with atomic operations in the current pipeline. Future work will focus on refactoring these kernels to support atomics in an asynchronous environment, aiming to extend the 1.33x speedup across the entire processing pipeline.
Summary updated