TL;DR
This paper introduces sRDMA, a novel protocol that brings efficient authentication and encryption to Remote Direct Memory Access (RDMA) by leveraging Network Interface Cards (NICs). By offloading cryptographic operations to the NIC and employing clever key management, sRDMA provides strong security guarantees without sacrificing the high performance that makes RDMA attractive for data-intensive applications.
What Problem Does This Paper Address?
Modern data centers heavily rely on RDMA for high-performance, low-latency data transfers. However, current RDMA implementations lack robust security features, leaving them vulnerable to various attacks. Adversaries can eavesdrop on sensitive data, tamper with memory, or even inject malicious code by exploiting the plaintext nature of RDMA traffic and its access tokens. While general-purpose security protocols like TLS or IPsec exist, they are not well-suited for RDMA due to their overhead and incompatibility with RDMA's one-sided communication model. This paper addresses the critical need for a security solution that integrates seamlessly with RDMA, providing strong protection without negating its performance benefits.
Methods
The core of sRDMA is a new transport type called Secure Reliable Connection (SRC) Queue Pair (QP). This design introduces two main innovations:
- Symmetric Cryptography on NICs: sRDMA utilizes symmetric cryptography for authentication and encryption, which is significantly faster than asymmetric methods. Crucially, these cryptographic operations are offloaded to the NIC, preventing them from impacting the host CPU and maintaining RDMA's low-latency advantage.
- Efficient Key Management: To mitigate the memory overhead on NICs, sRDMA introduces Protection Domain (PD)-level keys. Instead of storing a unique key for each QP, a single key is assigned to a PD, and QP-level keys are derived on-the-fly. This significantly reduces memory footprint. Furthermore, extended memory protection mechanisms are introduced, allowing for granular access control and sub-delegation of memory access without direct communication with the target host.
The protocol also cleverly reuses the Packet Sequence Number (PSN) as a per-packet nonce, avoiding the overhead of transmitting separate nonces. This is achieved by extending the PSN counters to 64 bits on the NIC.
Datasets
Not discussed in the paper. The evaluation uses artificial and real-world traces, and a modified key-value store (HERD).
Key Insights and Ideas
- NICs are the Key: The paper strongly advocates for offloading security to the NIC. This is a crucial insight because it allows for high-performance security without burdening the host CPU, which is the primary advantage of RDMA.
- PD-Level Keys are a Game Changer: The memory constraints on NICs are a significant bottleneck for security. The PD-level key management elegantly solves this by drastically reducing the number of keys that need to be stored, making robust security feasible.
- Reusing Existing Information: The clever use of the PSN as a nonce is a prime example of optimizing within existing protocols. It demonstrates how to gain security benefits by understanding and leveraging the inherent properties of the underlying transport.
- Security Doesn't Have to Kill Performance: The evaluation results show that while there is an overhead, sRDMA can achieve near line-rate performance in many scenarios, proving that high-performance RDMA and strong security are not mutually exclusive.
Key Results
- Header Authentication: sRDMA can achieve line rate (20 Gbit/s) for header authentication with multiple threads, even with complex algorithms like SHA512.
- Packet Authentication and Encryption (AEAD): For full packet security, performance is impacted by payload size. AES128 achieves up to 19.6 Gbit/s (95% of line rate) for writes. Reads are generally slower than writes, with the highest bandwidth around 16.71 Gbit/s.
- Latency: Header authentication adds a small overhead (around 0.9 µs for writes). Full packet security (AEAD) incurs higher latency, especially with larger payloads, but remains within acceptable bounds for many high-performance applications.
- Key Derivation Overhead: The PD-level key derivation adds some latency, particularly when cache misses occur, but it's a necessary trade-off for reduced memory footprint.
- HERD Key-Value Store: Integrating sRDMA into the HERD key-value store showed a performance decrease of up to 21% in the PD-prot mode (without caching) and 7.3% in AEAD mode, demonstrating the impact of security on application-level performance.
Limitations and Future Work
- SmartNIC Implementation Overhead: The software implementation on the SmartNIC incurs significant data movement overheads. The authors suggest that a different architecture using specialized packet processing units or hardening into fixed logic could further optimize performance.
- Cache Dependency: The performance of PD-prot mode is highly dependent on the RNIC's ability to cache derived keys. Without caching, performance can degrade significantly.
- Sub-Delegation Limitations: The current sub-delegation mechanism restricts delegation to powers of two. While a segment tree approach could overcome this, it would require exchanging more keys.
- Future Work: The authors suggest exploring optimized datapath architectures, hardening sRDMA into fixed logic, and further research into more efficient key management and delegation schemes. They also highlight the potential for sRDMA to be integrated into other RDMA architectures beyond InfiniBand and RoCE.
Summary updated