Performance Evaluation and Modeling of Cryptographic Libraries for MPI Communications

Abu Naser, Mehran Sadeghi Lahijani, Cong Wu, Mohsen Gavahi, Viet Tung Hoang, Zhi Wang, Xin Yuan

Abstract

In order for High-Performance Computing (HPC) applications with data security requirements to execute in the public cloud, the cloud infrastructure must ensure the privacy and integrity of data. To meet this goal, we consider incorporating encryption in the Message Passing Interface (MPI) library. We empirically evaluate four contemporary cryptographic libraries, OpenSSL, BoringSSL, Libsodium, and CryptoPP using micro-benchmarks and NAS parallel benchmarks on two different networking technologies, 10Gbps Ethernet and 40Gbps InfiniBand. We also develop accurate models that allow us to reason about the performance of encrypted MPI communication in different situations and give guidance on how to improve encrypted MPI performance.

Introduction

High-Performance Computing (HPC) has traditionally prioritized speed over security, often operating in "trusted" private clusters. However, as HPC workloads migrate to the public cloud, data privacy and integrity are no longer guaranteed by the physical environment. Protecting sensitive data—such as medical, financial, or engineering records—becomes a critical requirement for cloud-based HPC.

Existing attempts to secure the Message Passing Interface (MPI) have often been flawed, using weak encryption modes like ECB that leak patterns or failing to provide data integrity. While modern CPUs feature hardware acceleration (e.g., Intel's AES-NI), the bottleneck has shifted: the time spent encrypting and decrypting at the endpoints can now exceed the time spent transmitting data over the network. This paper seeks to quantify this "crypto-tax" and provides a mathematical framework to predict these overheads.

Methods and Implementation

The researchers developed two encrypted MPI prototypes based on MPICH (for Ethernet) and MVAPICH2 (for InfiniBand). They integrated four popular cryptographic libraries: OpenSSL, BoringSSL, Libsodium, and CryptoPP.

The encryption and decryption process integrated into the MPI sender and receiver workflow.

A key technical choice was the use of AES-GCM (Galois/Counter Mode), which provides both privacy (encryption) and integrity (authentication) in a single, efficient pass. The study also extends the classic Hockney Model into an "Enhanced Hockney Model," incorporating library-specific encryption/decryption rates to predict performance across different hardware environments without needing to run the full encrypted MPI stack.

Key Insights: The Impact of Concurrency

A critical finding of the study is that while encryption overhead appears daunting in single-pair "ping-pong" tests—showing up to a 215% increase in latency on InfiniBand—the impact is significantly mitigated in real-world parallel applications.

Bandwidth comparison across different message sizes and number of communicating pairs, showing how concurrency reduces relative overhead.

As shown in the multi-pair tests, the overhead drops as the number of communicating pairs increases. This is because encryption and decryption tasks can overlap with network communication and effectively utilize multiple CPU cores. In a 64-process NAS Parallel Benchmark run, the average overhead for BoringSSL was a manageable 12.75% on Ethernet and 17.93% on InfiniBand.

Performance Results

  • Library Comparison: OpenSSL and BoringSSL (Google's fork) emerged as performance leaders, significantly outperforming Libsodium and CryptoPP in HPC contexts.
  • Network Sensitivity: Encryption overhead is more pronounced on faster networks. On 40Gbps InfiniBand, the high network speed makes the CPU a massive bottleneck during encryption. On slower 10Gbps Ethernet, the network itself remains the primary bottleneck, making the relative cost of security appear smaller.
  • Model Accuracy: The "Enhanced Hockney Model" closely matched actual benchmark results, providing a reliable tool for researchers to estimate security costs on different hardware configurations.

Conclusion and Future Work

The study demonstrates that "security by design" is feasible for HPC with acceptable overheads, provided high-performance libraries like BoringSSL are used and concurrency is leveraged. Future work will address scalable key management and "streaming encryption" techniques—pipelining encryption with transmission to further hide overhead.

Summary updated

Abstract

In order for High-Performance Computing (HPC) applications with data security requirements to execute in the public cloud, the cloud infrastructure must ensure the privacy and integrity of data. To meet this goal, we consider incorporating encryption in the Message Passing Interface (MPI) library. We empirically evaluate four contemporary cryptographic libraries, OpenSSL, BoringSSL, Libsodium, and CryptoPP using micro-benchmarks and NAS parallel benchmarks on two different networking technologies, 10Gbps Ethernet and 40Gbps InfiniBand. We also develop accurate models that allow us to reason about the performance of encrypted MPI communication in different situations and give guidance on how to improve encrypted MPI performance.
BibTeX
@misc{naser2020performance,
  author = {Abu Naser and Mehran Sadeghi Lahijani and Cong Wu and Mohsen Gavahi and Viet Tung Hoang and Zhi Wang and Xin Yuan},
  title = {{Performance Evaluation and Modeling of Cryptographic Libraries for MPI Communications}},
  year = {2020},
  eprint = {2010.06139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.06139},
}

Loading related content...

Figure