TL;DR
This paper introduces an adaptive method for selecting InfiniBand transport services (RC vs. DC) within MPI to optimize performance and manage memory. By dynamically analyzing communication patterns, their approach achieves significant latency reductions (13-19%) compared to static methods, making large-scale distributed computing more efficient.
What Problem Does This Paper Address?
High-Performance Computing (HPC) clusters heavily rely on InfiniBand (IB) for high-speed networking and MPI for parallel programming. IB offers two primary transport services: Reliable Connection (RC) and Dynamically Connected (DC). RC offers lower latency but consumes substantial memory, especially with many connections. DC is more memory-efficient but can suffer performance degradation under certain communication patterns, like many-to-one or one-to-many communication bursts. Existing MPI libraries often use a single transport service or a static selection, failing to adapt to the dynamic communication needs of applications. This leads to suboptimal performance and inefficient resource utilization. The challenge is to intelligently select between RC and DC for each communication pair to balance latency and memory consumption, adapting to the application's evolving communication patterns.
Methods
The core of this work is an adaptive transport service selection mechanism called Adaptive Queue Pair Selection (AQPS). It operates in four main components:
- Software Message Channels: AQPS defines three software message channels built on top of IB transport services:
- DC-sendrecv: Uses DC QPs for send/receive operations.
- DC-RDMA: Uses DC QPs for RDMA (Remote Direct Memory Access), offering lower latency than DC-sendrecv.
- RC-RDMA: Uses RC QPs for RDMA, generally the fastest but most memory-intensive.
- Finding the Best Mapping: This component analyzes communication statistics gathered over a historical time window. It uses a combinatorial optimization algorithm to find the optimal mapping of RC-RDMA and DC-RDMA channels to source-destination pairs. The algorithm aims to:
- Prioritize RC-RDMA for performance where beneficial (e.g., small number of source ranks, high message counts).
- Offload communication to DC-RDMA to limit the number of active RC-QPs, preventing RC's performance degradation due to HCA cache limitations.
- Ensure distinct channels for destination ranks to avoid contention.
- Limit overall RC-QP usage to manage memory.
- Establishing Software Message Channels: This component dynamically sets up the selected RC-RDMA or DC-RDMA channels. It involves a background thread that suspends MPI send operations, prepares necessary buffers, and initiates connection requests to establish the chosen transport service.
- Evaluating Different Mappings: This component continuously monitors communication patterns. It periodically re-evaluates the best mapping using updated statistics and decides whether to switch to a new mapping if it offers significant latency benefits. This allows AQPS to adapt to changes in the application's communication behavior over time.
Datasets
The paper evaluates its method using micro-benchmarks designed to represent problematic communication patterns for IB transport services:
- Onetoall: One process sends messages to all other processes.
- Alltoone: Many processes send messages to a single process.
- Repeat: Alternates between onetoall and alltoone patterns to test adaptability.
These benchmarks are run with varying numbers of processes (8, 16, 32, 64) on a cluster environment.
Key Insights and Ideas
- Dynamic Adaptation is Crucial: The paper strongly demonstrates that static transport service selection is insufficient. Applications' communication patterns change, and an adaptive approach is necessary to maintain optimal performance.
- RC vs. DC Trade-offs are Complex: Neither RC nor DC is universally superior. RC offers speed but at a memory cost, while DC is memory-friendly but can be slower. The optimal choice depends heavily on the communication pattern and scale.
- HCA Cache Limits RC: A key insight is that RC's performance degrades when the number of active QPs exceeds the InfiniBand Host Channel Adapter's (HCA) cache capacity. AQPS directly addresses this by limiting RC-QP usage.
- DC Connection Overhead is a Bottleneck: DC's dynamic connection setup and teardown can introduce significant overhead, especially when many QPs need to communicate with different peers or when multiple QPs contend for the same destination. AQPS aims to mitigate this by strategically using RC-QPs.
- Latency Benefits Outweigh Adaptation Overhead: The paper shows that the latency gains achieved by AQPS significantly outweigh the computational overhead of analyzing communication statistics and adapting the transport service selection.
Key Results
- Latency Reduction: AQPS achieved latency reductions of 13%-19% in micro-benchmarks compared to static methods like DC-RDMA-OD (which primarily uses DC-RDMA) and RC-RDMA-OD (which primarily uses RC-RDMA).
- Memory Scalability: AQPS demonstrates better memory scalability than RC-RDMA-OD, especially at larger process counts. While RC-RDMA-OD's memory consumption grows linearly with the number of processes due to RC-QP creation, AQPS limits RC-QP usage and employs a scalable statistics table.
- Adaptability: The "repeat" micro-benchmark showed that AQPS effectively adapts to changing communication patterns, achieving a 21-24% execution time reduction compared to static methods.
Limitations and Future Work
- Parameter Tuning: The performance of AQPS relies on carefully tuned parameters (e.g., time window size, QP pool size, thresholds). These parameters were determined through measurement and estimation, but their optimal values might vary across different applications and hardware.
- Other IB Transport Services: The current work focuses on RC and DC. The authors suggest future work could explore incorporating other InfiniBand transport services like Unreliable Datagram (UD) and Extended RC (XRC) to potentially offer further benefits.
- Micro-benchmark Focus: While the micro-benchmarks represent problematic patterns, evaluating AQPS on a wider range of real-world HPC applications would provide a more comprehensive understanding of its effectiveness.
Summary updated