CryptMPI: A Fast Encrypted MPI Library

Abu Naser, Cong Wu, Mehran Sadeghi Lahijani, Mohsen Gavahi, Viet Tung Hoang, Zhi Wang, Xin Yuan

Abstract

The cloud infrastructure must provide security for High-Performance Computing (HPC) applications of sensitive data to execute in such an environment. However, supporting security in the communication infrastructure of today's public cloud is challenging, because current networks for data centers are so fast that adding encryption can incur very significant overheads. In this work, we introduce CryptMPI, a high performance encrypted MPI library that supports communication with both integrity and privacy. We present the techniques in CryptMPI and report our benchmarking results using micro-benchmarks and NAS parallel benchmarks. The evaluation results indicate that the aforementioned techniques are effective in improving the performance of encrypted communication.

Introduction

As High-Performance Computing (HPC) workloads migrate from isolated, trusted clusters to public cloud environments, the security of sensitive data—ranging from medical records to proprietary research—becomes paramount. Protecting this data from eavesdropping and tampering requires robust encryption. However, a significant performance gap has emerged: modern data center networks, such as 100Gbps InfiniBand, are now faster than the single-core CPU capacity required to encrypt the data they carry. This "encryption bottleneck" can cripple application performance if not addressed with specialized techniques.

The Encryption Bottleneck

Traditional security solutions like IPSec or "naive" MPI encryption (which encrypts a message in its entirety before transmission) are ill-suited for HPC. Naive encryption is strictly sequential: the sender encrypts, then sends; the receiver receives, then decrypts. This forces the receiver to sit idle during the sender's encryption phase, creating massive overhead. On high-speed links, a single CPU core cannot keep the network "pipe" full, leading to overheads exceeding 400% for large messages.

CryptMPI Design and Methodology

CryptMPI addresses these challenges by overlapping computation with communication through several novel strategies:

  • AES-GCM for Privacy and Integrity: CryptMPI utilizes AES-GCM, providing both confidentiality and authenticity. It leverages hardware acceleration via Intel AES-NI to minimize per-byte costs.
  • The (k, t)-Chopping Algorithm: Instead of treating a message as a single block, CryptMPI chops it into segments. It employs $t$ threads to encrypt segments simultaneously and pipelines the transmission. This allows the receiver to begin decrypting the first chunk while the sender is still processing subsequent ones.
  • Exploiting CPU Surplus: Modern HPC nodes often have idle CPU cycles (e.g., via Hyper-Threading) that the primary application does not fully utilize. CryptMPI "steals" these cycles to perform parallel encryption without significantly impacting the main workload.
  • Automated Performance Modeling: The library uses a mathematical model based on the Hockney communication model to dynamically select the optimal number of threads and chunks at runtime, switching to the chopping method only when messages exceed 64KB.

Performance Evaluation

CryptMPI was evaluated using micro-benchmarks and the NAS Parallel Benchmarks (NPB) on high-speed clusters. The results demonstrate that CryptMPI effectively hides encryption costs behind network transmission time.

Figure 6: Bandwidth comparison between Unencrypted, CryptMPI, and Naive approaches across various message sizes.

As shown in Figure 6, the naive approach's bandwidth plateaus quickly, while CryptMPI scales significantly better, approaching unencrypted performance levels for large messages. For a 4MB message, CryptMPI reduced the naive approach's 412.4% overhead to just 13.3%.

Figure 2: Bandwidth performance of Unencrypted, CryptMPI, and IPSec for multiple communicating pairs.

Furthermore, CryptMPI maintains high aggregate throughput in multi-pair communication scenarios. While IPSec performance remains flat and low due to the sequentialization of encrypted flows, CryptMPI scales effectively with the number of communicating pairs (Figure 2). In real-world applications like the communication-heavy NAS "CG" benchmark, CryptMPI's inter-node overhead was limited to 12.3%, a stark contrast to the 79% overhead seen with naive encryption.

Conclusion

CryptMPI demonstrates that high-performance encryption in HPC is achievable by moving away from sequential processing. By combining multi-threading, message chopping, and dynamic performance modeling, CryptMPI provides a secure communication layer that meets the demands of modern, high-speed cloud networks. Future work will focus on extending these optimization techniques to collective MPI operations.

Summary updated

Abstract

The cloud infrastructure must provide security for High-Performance Computing (HPC) applications of sensitive data to execute in such an environment. However, supporting security in the communication infrastructure of today's public cloud is challenging, because current networks for data centers are so fast that adding encryption can incur very significant overheads. In this work, we introduce CryptMPI, a high performance encrypted MPI library that supports communication with both integrity and privacy. We present the techniques in CryptMPI and report our benchmarking results using micro-benchmarks and NAS parallel benchmarks. The evaluation results indicate that the aforementioned techniques are effective in improving the performance of encrypted communication.
BibTeX
@misc{naser2020cryptmpi,
  author = {Abu Naser and Cong Wu and Mehran Sadeghi Lahijani and Mohsen Gavahi and Viet Tung Hoang and Zhi Wang and Xin Yuan},
  title = {{CryptMPI: A Fast Encrypted MPI Library}},
  year = {2020},
  eprint = {2010.06471},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.06471},
}
Figure