TL;DR
This paper tackles the latency overhead in MPI's rendezvous protocols, crucial for high-performance computing (HPC). They propose two novel protocols, GET and PUT, that significantly reduce communication latency by minimizing costly processor-device interactions and redundant control messages, leading to faster execution times in HPC applications.
What Problem Does This Paper Address?
High-performance computing (HPC) clusters often rely on commodity PC servers connected by high-speed networks like InfiniBand, which supports RDMA (Remote Direct Memory Access) for low-latency data transfers. A key component for large message communication in HPC is the Message Passing Interface (MPI) protocol, specifically its "rendezvous" mechanism. However, existing rendezvous protocols suffer from two main sources of latency:
- Processor-Device Communication: When a Host Channel Adapter (HCA) completes an RDMA transfer or needs to notify the processor, it involves costly communication over the PCI bus. This happens multiple times during a rendezvous.
- Extra Control Messages: In certain scenarios, particularly with receiver-initiated protocols, the system needs to send additional control messages to ensure data integrity, especially when the data being sent might coincidentally match a pre-set value in the receive buffer. These messages are amplified in many-core environments where multiple cores share an HCA.
These latencies, though small individually, accumulate and significantly impact the overall performance of HPC applications. This paper aims to reduce these specific latency sources within MPI rendezvous protocols.
Methods
The authors propose two improved MPI rendezvous protocols:
- GET Protocol (Improved Sender-Initiate): This protocol addresses the processor-device communication overhead in sender-initiated rendezvous. Instead of relying on a dedicated control message from the HCA to the processor to signal RDMA completion (which involves PCI bus communication), the GET protocol leverages the fact that an RDMA transfer modifies the receive buffer. The receiver now polls a specific memory location (the last byte of the receive buffer) for changes. This polling eliminates the costly processor-device communication for completion detection.
- PUT Protocol (Improved Receiver-Initiate): This protocol targets the issue of extra control messages in receiver-initiated rendezvous. The conventional receiver-initiated protocol (PUTNR) requires sending an extra control message if the value being sent by the sender happens to match a pre-set "constant" value in the receive buffer's designated polling location. This scenario can occur frequently if the constant value is common in the data. The PUT protocol mitigates this by randomizing the initial value placed in the receive buffer's polling location. This randomization drastically reduces the probability of a data collision, thereby minimizing the need for extra control messages.
Datasets
The paper does not use traditional "datasets" in the machine learning sense. Instead, it relies on:
- Micro-benchmarks:
- BOWTIE: A custom micro-benchmark designed to measure communication latency for various message sizes. It simulates bidirectional communication between process pairs.
- MPI_Alltoall micro-benchmark: Another custom benchmark to evaluate performance for the MPI_Alltoall collective operation.
- NAS Parallel Benchmarks (NPB) 3.3.1: A standard suite of benchmark programs used to evaluate the performance of parallel applications. The authors used Class W inputs for six NPB programs (cg, sp, ft, dt, lu, bt).
The authors implemented their protocols as a plug-in module for MPICH 3.0.1.
Key Insights and Ideas
- Polling for Completion is Viable: The GET protocol demonstrates that polling a memory location for changes (instead of relying solely on explicit HCA-to-processor notifications) can effectively replace costly processor-device communication for RDMA completion detection, at least for certain parts of the process.
- Randomization as a Latency Reduction Strategy: The PUT protocol highlights the power of randomization in avoiding worst-case scenarios. By randomly initializing a critical memory slot, the probability of needing extra, latency-inducing control messages is significantly reduced.
- Many-Core Amplifies Latency: The paper emphasizes that the latency overhead of control messages is multiplied in many-core environments due to shared HCAs and increased contention, making protocol optimizations even more critical.
- PCIe Bandwidth vs. Latency: While the GET protocol eliminates one type of processor-device communication, the improvement is limited because the HCA's inherent PCIe command issuance still introduces latency. This suggests that true latency reduction requires addressing both software protocol and hardware interaction aspects.
Key Results
- GET Protocol: Achieved comparable latency to the conventional GETCQE protocol, with the primary benefit being the elimination of one processor-device communication. The performance improvement was limited due to PCIe command overhead.
- PUT Protocol:
- Reduced latency by up to 24.60% compared to the conventional PUTNR protocol in worst-case scenarios (using the BOWTIE benchmark).
- Reduced latency by up to 16.37% compared to the GET protocol in the BOWTIE benchmark, and up to 26.13% in the MPI_Alltoall benchmark, attributed to fewer control messages.
- Achieved up to 11.14% reduction in overall execution time on NAS Parallel Benchmarks.
- Outperformed MVAPICH2-1.9b by up to 37.84% (PUT) and 25.67% (GET) in latency, primarily due to fewer control message transfers.
Limitations and Future Work
- GET Protocol's Limited Improvement: The GET protocol's performance gain was constrained by the inherent latency of HCA's PCIe command issuance. Further improvements might require hardware-level changes or more sophisticated polling strategies.
- Worst-Case Scenario for PUT: The PUT protocol's evaluation focused on the worst-case scenario for PUTNR. While effective, its benefit might be less pronounced in scenarios where data collisions are already rare.
- Future Work: The paper implicitly suggests that further research could explore hybrid approaches, more advanced polling mechanisms, or hardware-software co-design to further minimize latency in RDMA-based communication. The authors also mention the potential for switching protocols based on timing differences, though their proposed GET and PUT protocols are standalone improvements.
Summary updated